If you're already capable of riding in the drops on a road bike, or in a semi-aero tuck, and can maintain a comfortable 18-20 mph pace on level ground, then maybe drag from your clothing matters.

If you're riding a hybrid or MTB, or as you've said, riding ~12 mph in a somewhat hilly way to work, the aerodynamic drag of your clothing isn't a factor. Want to get faster?

Don't worry about clothing, worry about the motor and posture on the bike. But, mostly the motor.