I had the same problem when my bike was new. I eventually noticed it swerved toward the same side. I checked that I was sitting square and centered over the pedals. When I was convinced that was fine, I began checking the wheel alignment and then found one of the wheels was misaligned. This I checked with a carpenter's level to find when one wheel was perfectly plumb, the other wasn't and was offset to one side by about 3mm. Fixed the problem by filing (very, very carefully) the dropouts with a rat tail file. I've since concluded the fork has a bit too much offset and the fix for that is a new fork, eventually but no hurry.