I nearly always wear the lycra. Bib shorts and jerseys. But when I'm riding, it's to be riding; not to get somewhere. Except work. And then I change clothes when I get there.

In the earlier threads that dealt with this very topic, there were lots of links to sites selling hipster-approved cycling apparel. Some of it quite nice, particularly the colder weather stuff.