It is only a direct correlation if the two objects are of the same shape/external dimensions.

When dealing with air resistance, differences in shape have the potential to cause more of an impact on speed than differences in weight. My weight would increase if I were to carry an open umbrella on my shoulder as I coasted my bike down a hill, but it's quite clear I would go slower without the umbrella with me!

Don't you see the game that guy is playing with you?