Both my road bikes are 51cm and as such they were aquired not for frame size but for Top tube length and seat angle. I seem to like a TT length of 535mm and seat angle of 73/74. With those two measurements I can get the saddle in the right position and reach to the bars will be right. But I can ride and set up my Son-in-Laws 54 and in fact it was bought for me initially as a cheap bike.
The blue bike is the 54 and the white one is the 51. The black one is the 49 that had to be adapted to fit me.
My first road bike was a 49 and I rode that for a year as my introduction into road riding. But the rule of why the first bike being bought did apply. It was only there to tell me what my second bike should be. It was too small and I never really got comfortable on the thing. Second bike and 3rd bike and I knew the reach I wanted- could get the saddle position right and also knew the quality I wanted. I had not ridden the first bike for a few years but as I was selling it- I had to check it out for the new buyer and I just don't know how I managed to do so much on it. That frame is just a bit too small and I am cramped up on it. Back ache came in fairly quickly and even a bit of hand and neck ache. My Sils bike does have a TT that is only a few MM longer than my bikes and as such can be set up for me. I have to reach for the bars a bit but that could be adapted with a shorter stem and I can get the saddle in the right position.
My feeling is that frame size is not important. In fact if you look at the geometry of one particular size of frame from maker to maker- you will find differences in any case. What does matter is that you have to be able to get the saddle in the right position in relation to the pedals and those bars have to be at the right height and distance away from you. If you can manage that then the bike will be good for you.
But you only have one body that does not change much from week to week. There will be a Frame geometry and size that will suit you perfectly. Once that frame has been set up for you- it will be your fit. Change a couple of things from that perfect frame and you will have to make compromises and they can eventually come back and hit you.