the force of gravity difference is much more significant than the difference in coefficient of drag.

let's say you're 100 kilos and he's 80 kilos. your weight in newtonws would be 980, his would be 800. now say you're going down a 45 degree grade. that would mean that half of gravity is pulling you down the slope, while the other half is working to keep you and your wheels stuck on the slope. That means gravity would be pulling your bike forward with 90N more force than your lightweight friend.

here's another example..... a big ping pong (high drag, low mass) will still travel more slowly than a small steel ball (low drag, high mass)