Sizing would be more closely correlated to fit, and fit = speed.

The only thing size would affect is the weight of the bike, maybe geometry.

John Cobb has had good success with actually opening people up by make the fit more aggressive, and gaining watts that way.

All that's more scientific than I understand.

I ride a 56, but climb better, I'm sure, on a 54. Of this I have no doubt.
I ride a 56, but some long stretches feel better on a 58. Faster? doubt it.
I ride a 56, and I wouldn't want to descend at speed, on either a 54 or a 58; just no confidence.

Since I changed my overall fit, on every bike, to a more aggressive position, I think my performance is better.
Or I've convinced myself so.

Shortly, I'll be taking the next step, which is up a bit on the saddle, and down more on the bars.
Probably close to a 9-10cm drop.

Unless it makes my butt look bigger.