Ah, right!

I meant the frame. I'll give you two reasons.

First, because I think this is important: In my experience, when I'm caught between two frame sizes, I'll find even if the smaller frame size may give me a bit of a pain in the neck, it's actually more fun to ride than the bigger one. I've experienced this with two pairs of bikes, where I've done a lot of riding on both the 58 and a 60 cm versions of basically the same bike. Long story. I am guessing if you bought both bikes, you'd find you would ride the smaller one more.

Second, this is obvious but worth mentioning anyway. Your question about the stem demonstrates how it's pretty easy to modify a small bike to fit a taller rider. Change the stem, the handlebar, the seat post, even the crank arms, and you can dial in the fit pretty perfectly. A bigger frame will necessarily limit the adjustments you can make. If it turns out that the fit is not ideal right out of the box, your options for adjustments are limited.