I'm in a similar position. I've got significantly more drop than what I used to (still not much compared to some), and I've found that taking it slow helps. My spacers are all 5mm, and when I first started experimenting with removing them for more drop, I just moved one at a time and then rode it for a week or so to see how I liked it. If I wanted more, I'd move another one. I feel pretty comfortable with where it is now, though I could still drop it more if I wanted.

For me, though, I'm not going to cut the steerer tube in the forceable future, though I definitely could. The amount of drop I have now is comfortable for my average ride (2 - 3 hours), but this summer I'm riding across the country, and I want to be able to bump the handlebars up if 5-8 hours in the saddle every day proves to be a bit too much.

All that to say, I'd leave a little extra room just incase. It's easy enough to move the spacers, so long as you tighten the cap before the stem, like everyone has said. Unless you have an ungodly amount of tube above the stem, you're OCD, or you're racing, I say leave as much as you can.