Get some wool performance tshirts and socks. You don't have to wash them as often. Works great as a base layer or shirt.
When you get to work you can drape it over your chair or hang it. It will be dry and funk free when you are ready to go home.

Maybe look at the discount bins for winter clothing..shell, thrift store wool sweater, I'm thinking San Diego rarely gets lower than the 50s?