I too have a red jersey. ( red & black). I purchased it to match my MTB which is black with red decals. Also have a red helmet with white & black stripes.
I guess around this town seeing a cyclist decked out in lycra and a jersy is such a shock to their system they are left speechless! Especially when they see my fat behind on the skinny road bike.....
Make sure you carry some pepper spray, just in case they decide to stop.
Ride on, ride proud, ride loud..