Take an intelligent approach when adjusting the height of the bars. Regardless of whether the stem angle is changed or spacers are removed, don't make a huge adjustment all at once and don't forget that you'll get about a 3mm increase in reach for each 10mm that the bars are lowered. If you've been using a +6 stem and flip to minus -6, it will drop the bars by nearly 2cm and increase the reach enough that you might need one size shorter stem. For most riders, it's too large a change to do at once, but I've done it, changing from a -6 to a -17 stem (increasing the drop from 9 to 11cm).