Right. Flipping the stem is a big change. Try moving one spacer at a time above the stem, riding for a while at each height. When you get to about the same height as you would with it flipped, move the spacers back and flip the stem.