One thing to do is relax. Bikes steer in large part according to shifts in the rider's weight. Shifts in weight are unavoidable when you are peddling. If you get tense, you will just increase the problem.

I have heard some sources claim that bikes are intrinsically unstable. Riding a bike takes constant minor corrections which soon become second nature. The Wright Brothers were cyclists. Unlike other people pursuing the problem of flight, the Wrights did not try to make a plane that was stable in flight. They realized from their cycling that a pilot could maintain stable flight by constant minor inputs.

In function, one can often find smooth roads with white lines painted near their margins. Just ride a bit to the left of the line and see how close you can come to riding in a straight line. Imagine that you are riding on a railroad track. You should be able to ride pretty dern straight in a fairly short period of time.