There are 2 things that you need to understand.

1) wind drag is a squared function of frontal cross sectional area.
2) Gravity is your friend and enemy, but mostly your enemy, great for going downhill, but not so good going up.

A larger biker will have more variation in speed because of wind and slope.