2 ideas here: one is, no spacers, get the right angle stem. other is, stems going the wrong way look ugly.
i left 40mm worth of spacers, because i am unable to fadiddle around with too many different stems (i own two), and because i switch handlebars and bar height more often than i change pants (bullhorns, flop and chops, road drops, track drops...). i say leave a whole mess of spacers and deal with the fact that it's mildly inconvenient to the eye. ride it for a while and get a feel for where you really want that sucker to be. try out a few different stems. then, when you feel like you're really not using that top 20-30mm or whatever, chop it off and re-set the starfangled nut. oh my god, starfangled nut banger is my favorite tool name ever.