Experience.

Knowing how you felt on a particular size frame for several thousand miles and then trying other size frames to see what that does for you.

Larger frame might keep you more upright or allow you to get more stretched out.

Smaller frame will generally let you get more drop from saddle to bars and with longer stem just as stretched out as the larger frame. You probably can't get as upright with out the stem being ridiculously angled up.

After riding excessively large frames (64, 60, 59cm) for over 45 years, I'm on a 56 cm Tarmac since this March. Very comfortable right out of the shop. The only thing I changed was putting short cranks on it. I'm 5' 11" with 34.5" legs.