the comments about frame and wheel alignment are all pertinent. if the bicycle's wheels are not rotating in the same plane (and there are many potential causes for this), then their individual gyroscopic effects will essentially compete with one another, and you will have to provide some sort of external moment relative to your plane of motion to restore things; that is, you have to lean to one side.

i'm not convinced that a sufficiently misaligned setup would necessarily be visible to the naked eye. do your best to align/straighten everything, but you may have to live with the lean (that is, find another way to impress the ladies).