I don't mind wearing bike shorts and jerseys (typically plain ones) for most of my tour. I actually like that I am instantly recognizable as the guy they saw on the loaded bike. That said I do like to get out of bike clothes in the evening. I tend to wear running shorts and a tech tee shirt when off of the bike for the day. If it is chilly I will add one or more of the following:
Pile shirt
windbreaker (rain/wind shirt)
tights
rain/wind pants
pile hat
down vest (used to use as pillow, but now usually leave home)

I used to take a pair of light nylon zip offs, now I just use my rain pants when long pants are required.