I was doing my twice year removal and re-greasing of seat post, stem and fork steerer tube so that they won't get stuck.

I put the fork back in and tightened the head set with the stem cap bolt, doing the tests to make sure it is tight, but not too tight. (I have been doing my own bike builds for years and was a shop mechanic for a time.) Then I tightened the stem bolts to 5 nm and did the test again, but I could feel a slight looseness, or movement thru the handle bars. Testing the headset by engaging the front brake was solid and there is no play in the fork stanchions and it is not overly tight. I loosened the stem bolts and the feel of looseness and movement were gone. Then I re-tightened the stem bolts and it was back. I tried a different stem with the same results; headset tightened correctly with the stem cap bolt and all is fine, tighten the stem bolts and the slight looseness, or movement thru the handle bars is back. (I checked and reinstalled the headset and it is fine and it is relatively new.)

Any ideas?