what material steerer tube?

if carbon, try the carbon friction paste. if its alu or steel, try roughing up the steerer tube slightly with sandpaper so the stem gets a better grip on the steerer tube.

sealed bearings or loose ball? if you dont have the right amount of preload, that will definitely cause problems, good luck.