Of course aero matters, a lot, but I think you are giving too much importance to your broad shoulders and his shaved legs.

First, just equipping your road bike with aero-bars would close most of the aerodynamic gap. Riding o the drops vs aero bars matters a lot more than than having slightly wider shoulders.

Secondly, the difference in your times was not just due to aerodynamics. Cyclists tend to think too much wind drag and forget about gravity. If your estimates are correct, you are talking about a 40 pound weight difference between you and the other guy. 40 pounds is like carrying two whole extra bicycles on your back. I bet that accounts for most of the time difference (if the road was hilly then I have no doubt about it).