Remove spacers first.

The average spacer is 5mm. Removing one spacer lowers your handlebar by about 5mm.

Flipping a 6 degree stem will change the overall angle by 12 degrees, which on a 100mm stem is almost 2cm. It often is best to make fit changes gradually.