When faced with this problem I usually rely upon clothing made with clothing made by a British company called "Rohan". Their range are ideal for your situation in that they are light, non-wrinkle and of very high quality. They are made to dry quickly also so are very suited to camping. I'm sure there must be similar hi-tech clothing made in the USA.

http://www.rohan.co.uk/mens-clothing