Is it the same tendency just wheeling your bike by hand? Eg. try and just push it along in a straight line by just pushing the bike from a hand on the saddle.