I changed my fork to a smaller size and now my fork is kind of wobbly. Especially when you hit bumps. Its nothing crazy but you can feel the fork slightly shift.

Is something to do with the new stem being larger than my old one so now the headset doesnt settle properly?

I tried loosening the pinch bolts, then lifting up and the fork and putting all the weight on the headset, as to have the fork reach a highest point as possible. Then tightening everthing up.
This did help a lot but it still didnt fix the problem entirely.

Im worried if I leave this it will become a lot worse problem.