It's more complicated than that, but most people can make a bike a size up or down fit them if they need to. But everyone is built differently. I have to concern myself with seat tube angle more than anything, but I'm just an odd duck.

Still, I've noticed that going up a size with a shorter stem or going down a size with a longer stem tends to make the bike feel different and handle differently.