You're going exactly where your attention is leading you.

Look at the edge of the road, and that's where you'll subconsciously move; pay attention to the traffic behind you, and you'll move towards them. Put both together and you've got a wobble.

Funny thing is, in marching band, it works the same way. I had a bass drummer trying to fit with a diagonal form by looking forward down the line and to his rear, and his intervals were just all screwy. I told him, "Forget about everything going on behind you, and just look down the line." Bam, just like that, he was arrow-straight through the move every time.