Like [MENTION=270877]timdow[/MENTION], I wear "normal" cycling shorts to keep chafing at bay. 20 miles one way is a long ride, and what works for a 3-5 mile commute, like khakis, often doesn't do as well on a longer ride. (I'll suggest that as you push 200 miles on a ride, you'll start to understand, and you may also want to investigate chamois cream to knock down any hot spots before they get raw.) If you have an office door clothes hanger, locker, or even a cube wall to hang bike clothes on, they'll usually dry before it's time to go home. (Exception: storm with an inch of rain coming in!)

When I was commuting regularly, and when I start again, I often left one pair of shoes at work, and used my second pannier to ferry another pair in and out as needed. First pannier carried clothes, pocket contents in a zip lock baggie. Lunch and a towel can fit in the clothing pannier, or they fit better in the shoe pannier if I'm taking it in that day.