I don't think there's a lot of difference in material between a swimsuit and cycling shorts. I would not be too concerned about fading. However, you might be more comfortable on the rest of your ride if you took a swimsuit so as not to have a wet chamois.