If the OP felt strongly enough about it, he could have opted to not participate in the ride, or withdraw from it. He can also vote with his bike by not participating in any others put on by that ride organiser.
But it is worth pointing out that the comment wasn't aimed at him because, on his own admission, he is no longer a Clyde... and the Clydes in the group seemed to enjoy the joke by laughing at the comments.