I wear roadie shorts under baggy shorts, or under whatever, esp if I have to stop at a store or something on the way home. Also helps to not freak people out with my "Spinal Tappishness" on the way from the bike rack to the changing area at work. I have roadie shorts because I become a roadie on some weekends. (Well, an underachieving roadie.) Same thing homebound as workbound, pretty much.
Exceptions galore depending on weather. If it's perfect out, I'll ride home in my work clothes to save time.
Some winter gear other than bottom-layer gets re-used a few times. I only have one pair of full-length tights, for instance.