Yes. This is true if you are riding solo. In fact, I have been tracking my time for a couple of years on a 40 mile return trip with 1700' of climbing which I do regularly on all three bikes...To my surprise, the average times for each of the three bikes are indistingushable within the variance. Thus, by observation, there are often a lot of random variables effecting average speed over a return course which will mask a few percent systematic speed reduction on the uphills.

Riding in a group is a different matter, though. As has been mentioned previously in this thread, the heavier bikes will struggle due to both slower acceleration and slowness on the hills and, being systematically and constantly left behind (except on downhills which don't weigh much in the average) will eventually miss the draft afforded by riding with a group.