Unless you are drinking heavily, I think it is more a matter of courage. When you were young, you never woried about falling off your bike. Now a simple fall on your bike can do some serious harm.
For that reason, you tense up, making it difficult to go straight. When you do veer a little, you quickly reach for the bars, again making a rapid weight shift, which makes you feel it was a good thing you grabbed your bars when you did.
Relax, unless you hit a rock or a big wind just, the chances of falling are slim.