Let's say I currently ride a 56 cm frame. If I find a killer deal on a 54, couldn't I buy it and then make up the frame size difference with changes in stem length and seat post height and/or setback?

This is hypothetical. I'm just curious how much the frame size matters for a good fit. Obviously a 52 and a 62 are quite different. But a couple of centimeters?

Thanks.