Is your stem jammed down low? If so, dabac may be right about jamming with the internally butted steerer. His wooden block stategy sounds right.
From Sheldon:

Stem not down low?
Whether the stem has a wedge or a conical jammer, either can set up weirdly when tightened.
How hard are those taps you're giving the stem bolt? Persuasive?
Maybe a proper hammer with a bit of wood over the bolt head to protect it.