Originally Posted by
rat fink
It matters more to keep one (5mm) spacer on the top of the stem, which is said to prevent the steer tube from cracking.
You also need some sort of a spacer (a cone spacer, or zero stack) under the stem to act as a bearing cap for the headset bearing.