I've worked my way down to taking off the last little spacer under my stem.

I have an FSA threadless headset. Pretty standard stuff. Is there any reason I need to have a spacer between the headset and the stem? I thought I remembered hearing this was recommended for some reason (maybe to help the bars turn easier or something), but I could be mistaken.