Unless of course one is not on a closed course, and then all those half MPH gains are lost waiting for lights and traffic.

Or, from my anecdotal experience: I did an organized ~30ish mile group ride with a dozen and a half folks a few weeks back, on a 40 year old bike pushing 35#. I finished in the group of four in the front, with two CF bikes and an older steel Bianchi. I am in no way a racer, or even all that fast, have little doubt the riders on the two CF bikes I finished with could have taken off and left me at will, but others on CF bikes also finished a few minutes behind me.

An "average" cyclist is not racing, is generally not on a closed course where minuscule advantages add up big, and IMO are generally limited far more by the engine than the frame.