If the play increases over time, and if you don't have any play immediately after adjusting the headset, then I would venture that the stem is slipping.
What type of stem is it? Some stems will seem tight but not be if the bolts need to be tightened alternately but aren't (Thomson Elite is a good example).
Do you have grease on the steerer tube where the stem clamps to it? This part of the steerer should be dry and free of grease.