I tend to not bother with road jerseys and bring wicking short and long sleeved shirts that can double as street clothes, the sort of thing joggers would wear. I actually don't bother with bibs when I tour either: I'll bring a couple of chamois liners and a pair of unlined mountain biking shorts which can double as streetwear (with normal underwear) when appropriate.

I cycle with spd mountain bike shoes, so I'll bring a lightweight pair of runners or casual shoes for off the bike. I'll bring a merino sweater and use my cycling jacket when off the bike if the weather dictates. Speaking of which, I will usually pack a rain jacket just in case.

I like to bring one pair of pants or jeans, something comfortable as well. I'll wash chamois liners & jerseys as necessary.