I ride a 54 and also a 56, both road bikes, and they both fit fine and are comfortable.

They are set up differently, for different positions, basically different riding styles. I think it's logical to start with that even before measuring an inseam. What riding style, how do you want to sit on the thing.

But it's about reach, and honestly if you wanted the same position 2 or 3 cm isn't that large an adjustment between the stem and setback.

Too large, in my humble opinion, is when the seat is too low to get the drop you want or when you're stretched awkwardly to ride on the brake hoods.