Yes, I know this thread is from a while back, but it's still applicable today, and might be interesting as we think about our upcoming tours. 
In answer to the question ... I've gone through a transition.
On my very first tour, I wore mtn bike shorts, a T-shirt, and runners, but at that time, I wasn't into wearing cycling gear for everyday riding either. Once I started wearing cycling gear for everyday riding, I also started touring in cycling gear ... until toward the end of my first Australian tour.
Queensland was so hot, and riding in cycling gear became extremely uncomfortable. Plus we were stopping at beaches along the way, so I started riding in a bathing suit and beach shorts. Thank goodness for my Brooks saddle!!
Since then I will usually wear cycling shorts, but not always, and I often wear a pair of basketball shorts or my zip-off pants over them. And I've got a small collection of wicking T-shirts which I like better than jerseys. I can wear them when I'm riding the bicycle, or when I'm sightseeing on foot, and look like a normal person.
In fact, my company amazed me the other day by giving me a really nice wicking golf shirt made by Ontour - a sports clothing company:
http://www.ontoursport.com/home.html for the work I did on a particular project.
When they told me they were giving the team T-shirts, I thought they'd be those generic boxy cotton T-shirts most companies and events give away ... you know the kind. A person could use them as a tent they're so big, and they're too unflattering to wear anywhere, so they end up in the back of the closet.
So I was very pleasantly surprised by the kind of T-shirt they did give me!! I can wear it for spinning classes, and I can also wear it on a tour.
And wool ... I love my merino wool top for touring.