If the headset is too tight it might. Is pretty obvious to know if its too tight so i wont go to discuss that and from what u say it is ok.

If the bike goes to one side or another IMO is the bike or your skills riding w/o hands. Some bikes do what you describe no matter how good you are at riding no hands. What bike are we talking about here?