Hills and rough road surfaces make it tough to maintain your momentum between pedal strokes.
If your pedaling style is slow and uneven (visualize that there is a ketchup packet on the pedals that you want to stomp and explode), it won't make much difference how fit you are.
The people who have a quicker cadence and can apply force more evenly for more of the pedal revolution will have an advantage. They are not subject to the same accelleration/decelleration a masher will endure.

The way I see it, on the flats and smooth surfaces you are limited by your strengths.
On hills and rough roads, you're limited by your weaknesses.