For a 3-4 mile commute.....I'd just layer some technical running/workout gear. The fabrics have come a LONG way lately. Get rid of the cotton.

Grapevine Mills Mall. Under Armor store (shop the racks in back ).

Or...Wally World has had their 'competitor' brand to UA (not too bad, just not as durable) on sale around our neighborhood (North DFW), cause they are stocking up winter wear.