Any chance the wheel(s) are not completely in the dropouts? Headset a little loose? Squeeze the front brake and rock the bike back and forth, any noise or play from the fork/handlebars would show up.
Tires at a reasonable pressure? Are they seated properly, or maybe not quite round? Tires that aren't built straight can cause the steering to swerve as the contact patch goes off center.