I don't think the clothes matter. Wear what makes you feel comfortable. If you're comfortable you'll be more confident on the roads and in my limited humble opinion that's what matters most.

Just my two cents now:
I ride roughly the same distance in roughly the same time wearing roughly the same clothes. At first browsing around here with many Herculean tales of epic commute distances in half my time in the worst rain/snow/hurricane/heat since records were kept made me feel pretty substandard.

But here's the deal: I ride through 5 stop signs and 18 stop lights. My flat-bar road bike is heavy even before I put me and my stuff on it. The sitting position is very upright, and I enjoy that, so I can look around and hopefully avoid death.

So if you want to ride faster perhaps get a different bike or ride fast on the weekends following a route that makes that possible?

all the best.