Look at how track racers are built. Lots of "bigger" guys there, at least in terms of power producing muscles.

As a smaller guy I do feel I have advantages in terms of tucking behind bigger guys, and on climbs or rolling hills. For descending I feel it's more about technique/tucking than weight, but I suck at it anyway. Also - riding into the wind and on flats at sustained higher speeds is tough for me.