I get this comment all the time. They just don't see to realize that you are exerting yourself when you ride and that keeps you warm. It surprises most people that even when it's below freezing outside I wear a thin merino wool shirt and a very lightweight jacket over the top. Anything more and I get too hot.

The other comment around here is when I ride on a rainy day (rainy days are a common winter occurance around here). "You rode in this rain?". I usually answer with a smile "yup, we are not made of sugar, you know", or "yes, humans are waterproof after all". That usually shuts them up...