Actually I don't dress up much either. My work requires the Dockers type casual look, so I just throw my outfit in my backpack and it looks fine when I change into it at work.
The biggest problem is that sometimes I forget to bring my boxers, and I have to wear the damn cycling shorts under my pants all day. What feels great on a bike is uncomfortable when worn all day at work. In fact, today was one of those forgetful days, and even now I'm sitting at the computer with dampish chamois and spandex bunched up between my legs.
