Some manufacturers specifically advise you to not leave any spacers above the stem except for temporarily while you are fitting the bike. The reason they give is that if you have spacers above the stem, then the part of the steerer the clamps down against will not be supported by the top cap. This might be just some cover-your-ass thing, but I know Cannondale and Specialized warn against it.

I assume you know this, but whatever you do, don't make the top your stem/spacers "flush" with the steerer tube. The top of the steerer tube should be 2-3 mm below the top of the stem/spacers.