I took everything apart and put it back together 5 times tonight and every time I do it , I still have the wobble. I tried changing the spacers around to different configurations and currently have it back the way I got it with the three spacers under the stem. Still wobbly.

Does anyone have any other suggestions? Is it just a minor annoyance that I can live with or should I be concerned about the integrity of the bike when I'm riding?