It's a bad plan to wear dirty shorts/bibs. Bacteria is the primary source of saddle sores, and washing gets rid of the bacteria.

I can't stand wearing dirty socks either. I assume you'd get athlete's foot if you do, but am not 100% sure on that.

As far as I can tell, other clothes can be re-worn without harm. Well, not to you, although it might not be good for those around you.