You can get cycling underwear, which you can wear under a pair of ordinary shorts. Most camping shops sell T shirts in wicking materials. They are usually a bit looser fit than stretchy cycling jerseys.
I do a lot of riding in fairly standard hiking gear. The material work as well as cycling gear, but Id feel a bit stupid riding to the library or shops in my shorts and yellow jersey.