To support my average speed assertion, i offer the following comparison:
A ride on essentially flat terrain will allow you to go a very consistent speed. Your average might even be close to the speed you see while actually riding.
But, a ride with a lot of hills will be made up of long slow ascents and descents that are equally long in distance, but much shorter in time. The time lost on the ascent is rarely regained on the equivalent descent. Hence such a ride will produce a lower average speed.
I'll leave it to you to judge which ride produced the harder workout.
But, of course, many of us frequently ride the same routes. Except for wind, the average can be a useful comparison here.
Cheers,
Jamie