By the way, if anyone can bring an EEOC/sexual harassment suit it is the OP. Imagine if a lady in a skirt was told by her male supervisor that she can't wear a skirt anymore because it's too distracting to him?
Anyhow, Sexual Harassment comes in two parts. First is classic quid pro quo. I.e. you will be promoted, granted preference etc. if you do (fill in the sexual act). That's pretty cut and dry.
The second is that conduct of a sexual nature is creating what a reasonable person would consider a hostile work environment. The reasonable person qualifier is key, since that is what a judge would look at, not at what your unreasonable co-workers think. If you worked in bike shorts all day perhaps a reasonable person would feel it a hostile work environment. Since you simply ride in bike clothing that will not pass the test.
Your co-workers have elected to assign sexual tension or meaning to your lycra shorts that you don't even wear all day, and that you utilize for utilitarian reasons. They have taken their wholly contrived complaints and meaning up the chain, and their mentality (and indignant oggling) has created a hostile work environment for YOU.
What do I know about this? As a Capt in the USAF I have to be up to speed on sexual harassment laws and EEOC issues, since under me are enlisted troops of both sexes.
__________________
Good night...and good luck