it's an interesting question. In general, you wouldn't want to mess with the seat just because of lowering the handlebars. However, if you move your seat forward, you probably want to raise it a little. If you are just removing 5mm of spacer, it's not going to be necessary to move the seat. Take the spacer off, put it above the stem, and try it. No harm done.

A few years back, I raised my stem a little. I had a 1" spacer, which I decided was way too much, so I took it out. The result was a much better position for me. I didn't move the seat at all