There could well be something off on your bike so that the wheels are not properly aligned causing a bias to one side or the other. I have a Specialized Secteur that did not seem to track well and pulled to one side. After a lot of thought and measuring, I found the front dropouts were not perfectly horizontal when the bike was plumb. One dropout was glued into the fork just a tiny bit higher than the other. I worked out that if one dropout was one half mm higher than the other, the rim would be 3 to 4 mm out of plumb. I was able to easily fix this with a round file to remove just a bit of material in the dropout. The bike can now be ridden no hands but the best part is that it tracks much better and seems nearly on rails when cornering.
This may not be the problem and solution on your bike. You could visually check that the wheels are centered within the frame. Possibly dish on the rear wheel could be off and the front wheel also may not be centered due to spoke tension being off.