Assuming a good wheelset and properly inflated tires, I believe the rolling resistance differences are negligible.

Weight is a different story. Twice the weight means twice the force.

Aerodynamics I think are a bit of a factor, but, I don't think they come close to the weight factor.

So, yes, fat dude will pwn skinny guy on the downhills, assuming he is a fit fat dude.

Of course, he will give it back, plus interest when it's time to climb, so, you ain't gonna see many XXL yellow jerseys.