"You're crazy", is a common response but I've never felt it has been intended in a negative way. It usually comes off as "wow" or a round-about compliment implying it is something the person doesn't feel they'd be able to do themselves.

I'm always much colder on days where I take the bus, than days where I ride the bike. For most people that seems counter intuitive, and they wouldn't believe it until they experienced it first hand. There is a leap of faith you need to take in beginning cold weather commuting.