I've never had anyone say anything derogatory to me while I've been riding a bicycle, hiking, or motorcycling, etc. Or at least not that I could hear...
However, this apparently is not so uncommon with male bicyclists that wear spandex. And no, I don't have anything against such riders like some people do; I'm just stating what I've heard over the last couple of decades of riding.