Sounds like something is out of whack. It could be frame/fork related, but it could also be the wheels - check that the dishing is even. I had a badly-dished rear wheel on an old three-speed and it wasn't obvious until I mounted it all the way back in the drop outs (with the chain off the chainwheel.)

edit - beat to the punch