Unless you're riding in the vacuum of space, I'm gonna go with gravity. Math isn't my forte by any means, but

Fnet = mg − fd(v)

where the total force accelerating the object downwards is the difference between the force of gravity and the drag force; in short, heavier things fall faster. The weight is (usually) enough to overcome the relative differential in drag-- that is, a lighter rider on a lighter bike produces more drag relative to the force gravity is exerting on them. In the real world, my rider + bike weight is about 50lbs greater than my wife's; unless she drafts me down a hill, I will just pull away from her the entire way down.