By removing spacers first, you are making things stiffer because you have less unsupported steerer tube between the stem and frame. It will make a noticable difference when sprinting or climbing out of the saddle. I have my stem set up on one bike with only1cm of spacers and a positive rise stem. It feels much stiffer than when I had 3cm of spacers and the stem flipped. The bars are in virtually the same position but have a better more hooked up feeling due to fewer spacers.