Bit of simple thinking here, don't take spare shorts. Why do you need another pair of shorts anyway.

I had the same problem with my towel until I met an English cycle tourist who was an avid swimmer and lived in a small ford van back home, and never used a towel, he said put your clothes on and you'll get dry. It's worked for me since and that was more than 5 years ago.