Comfort.
If you can ride for hours without being sore.
I have been told I ride frames to large for me. I can stand over them and have from no clearance to over an inch.
But I have long legs and arms, and I am all cramped up on smaller frames. So I buy what feels comfortable for me.
I have read it all and tried the ideal fit, which sometimes is far from ideal, but you need to find what works for you.