Have you ever been frost bitten? It's common for repeated light frost bites to add up. I'm not sure i would jump on the Raynard bandwagon, that is a fairly specific condition. Some people have poor hand insulation, it is interesting to note that many Inuit people and northern Europeans have higher levels of subcutaneous fat on their hands. It has also been noted that feeling cold and being cold aren't connected, many people are more sensitive to various degrees of cold others freeze with less warning.

Don't bother drinking alcohol in an attempt to warm up, it will make you feel warm for about 5 minutes then some of your precious core heat will be lost and that is hard to get back.