This really sounds like a headset that doesn't turn as effortlessly and smoothly as it needs to in order to allow proper self-centering of the front wheel.

Note that this could be the result of some sort of localized alignment issue from one cup's mounting to another, or from a bent steerer, or from a crown race seat in need of proper finish machining.

And also note that even if a headset turns effortlessly without a stem installed, tightening the stem quill bolt bulges the steer tube elastically which causes the steerer to shorten slightly.
This results in the headset adjustment tightening up perceptibly.

Perhaps also, the bearing races may have become slightly dented from riding on a tightly adjusted headset, which will make future adjustments somewhat of a compromise between too loose and too tight (just as with any of the aforementioned alignment issues).