Not true, on downhills the higher the gear the better. It is the uphills that determine the highest ratio you can have (or else you walk), the downhills (and your maximum cadence) that determine your minimum gear.

I find the fixed great on flat and gently rolling rides (Surly called theirs the "Steamroller" for a reason!); however, having to accelerate from a stop (eg. at lights) is slower, and downhills are slower for me too, as I am limited by cadence. A freewheel overcomes this limitation.

The gentle uphills that I ride fixed are not a problem in terms of speed.

I don't think there is any doubt, however, that on varied terrain you go faster overall on a geared bike.