if nothing is bent, it is 99% headset adjustment. if you start to go one way and can't straighten it out, your headset is probably too tight.

also, i guess it could be your saddle (type, adjustment) since you use that to steer.