Shouldn't change the other components of your fit much, if at all. But do it slowly, a little bit at a time. For example, move the smallest spacer above the stem. Ride for a while. Then move the next smallest spacer above the stem, putting the smallest back below if needed. Ride for week or two at each adjustment.
You'll figure out when you've gone to far.