It's been said but I'll reiterate:

Check your soaps- both your detergent and what you're washing with when you shower.

Rinse off well after showering.

Dry off well after showering.

Funny but the fact that you're so diligently washing the shorts and showering makes me suspect a soap allergy or reaction.