IMO, rider weight is a factor. Most heavier riders can generate more power than their lighter counterparts. However, this doesn't necessarily translate to faster speeds on the road because that additional power is also pulling more weight. For a rider weighing 120lbs, dropping a couple of pounds on the bike is probably going to be more effective than for someone at 200lbs. Also, I doubt there is much difference when not climbing or accelerating. If you are riding a flat route, I doubt there is much difference once up to speed.

From my experience, wind affects speed more than anything. Even with a very aero bike (even with Zipps), wind will still affect the rider's speed more than anything else. Of course, there is not much we can do about that.