I think it has something to do with the clothing. I've noticed it also, esp in professional cycling. Even in watching the olympics I am surprized when they show pictures of some of the athletes when they are not competing.
As for men being faster, I think a lot of it has to do with mental attitiude (I can say this cause I'm a chick!). There a exceptions but for the most part women don't try very hard in a lot of phyical activities. I've always ridden with guys because my female friends just wouldn't pedal!!! And some of these women were Marines!!! (I used to be married to a marine and lived in milatary towns). Working out at the gym is even worse! Females are so brainwashed by "womens" magazines and other "women" media that they don't even know how to try hard!! Take a simple excersize like machine shoulder presses, just as an example. They will start out with a certain light wt and a year later will almost be at the same wt! After 12 reps they could crank out a few more but hit a mentle block and stop!! I guess what I'm trying to say is that it's hard to judge wheather a female is slower because of being a female or just not reaching their potential because of the way they think. I'd rather ride with a group of fellas and be the slow one than ride with a bunch of women that aren't even trying! I've also noticed there is a curve, and at some point gains come very slowly. It's just the difference between women and men.
Its sad because in this day and age there are many good female athlete role models but the old mindset prevails. And no, I don't look masculine.
Sorry, this turned into a rant.