This certainly sounds like a pitted or brinnelled headset bearing race. The cheap solution is, provided it is a cup-and-cone serviceable headset, replace the bearing balls, but reassemble without the original cage holding the bearings in place and use one extra ball than would fit with the cage. This will put the new balls out of phase with the pits in the bearing races.
NB. this method is much more finicky as keeping the balls in place during assembly is a bit of a pain, but if you put a fair amound of nice thick grease on the bearing surfaces it will keep the balls in place as you work. It may be necessary to position the bike such that the steerer tube is completely vertical during assembly - this will help prevent the balls from moving around or trying to roll off into the head tube.