well, it is kinda creepy if you enter the office in spandex. I don't blame him for bringing that up. if you can't change outside, maybe at least throw on some loose-fitting jacket and shorts before entering the office?
or maybe he has seen cyclists before who become holier-than-thou because they are
1) saving the planet
2) getting exercise
3) raising money for the sick every weekend on their many ride-a-thons
as there are many of these about. so maybe he thinks you are one of those. maybe you have done something to make him think that way?