If you are doing 20 mph average on a course with stops and slow downs your doing better than most of us. The fact that there is only a few minutes difference between a hard effort and a not so hard effort is as MinnMan said - due to the ramp ups and downs and stop and go and also due to drag.
If you are hitting peak speeds above 25mph - which you must be given the description of the course, pushing an extra mile or two ph is a big effort because the power required to overcome drag increases with velocity to the third power, so small changes in speed make a big change in power when the speed is larger.