I think your mistake is assuming that bicycle specific clothes are necessarily about speed.

I love my riding fixed gear, but I wouldn't be able to keep up with a rider of comparable fitness using a road bike on an open course. Not even close. Don't kid yourself.