Have you ridden 40, 50 miles before? There's probably more harm to wearing bike shorts for 3,4 hours at a time (constant warmth and exposure) vs letting it air out for a bit. If you're paranoid, use your underwear for the ride back, or pack an extra set. You're not racing to beat the clock, and an extra 3-6oz is not going to kill you.