Coincidentally, the latest Fast Talk podcast is all about riding in the cold: Fast Talk podcast, ep. 35: How to train in the cold | VeloNews.com

One of the things they mention is individual variation -- some people just don't physically handle the cold well. I've always been kinda on the opposite end of the spectrum. Despite being skinny and having barely any insulation, cold doesn't bother me too much but heat just shuts me down.