I'm the cold wimp in my family (my wife doesn't ride), but I make myself ride all year, because I can't stand indoor trainers.

Also, I got a velomobile so it's much better than being in the open...still cold, but MUCH better. Coldest I've been riding in is ~7F.