Any of the younger riders around here would surely know better, being as there is some VERY fast company in the 55+ age group.
It would be hugely embarrasing if the wise-ass was to then get dropped for any reason.

And the rides here which have racers and casual/social riders assembling at the same start time/place are neatly segregated, with slightly different departures so that people don't lose track of which group they are riding with. The faster guys/gals almost always ride further, and the idea is usually that all might arrive back around the same time.

But I think such a comment would usually yield a good joking back-and-forth for all the possible entertainment, and all in good spirit.

I at 52yrs would have probably taken it as a compliment, and I don't mind the attention. The old bikes really get noticed these days, but not everyone says anything. Some sort of resent it, like I didn't spend enough to support our shops(?), while others seem hugely amused that a rider on an old, stock-looking bike can hang with the carbon crowd. Riders seem very divided this way.
As far as clothing, I wear bike apparel on group rides because it allows following riders in the pack as little visual obstruction as possible.
Off road, I usually use trousers when it's cooler, for greater warmth and protection.