The heat loss I'm talking about took many hours. I was plenty warm over my whole body, with the exception of thumbs, fingers, and feet. I conclude that my body heat was leaving those areas faster than it was reaching them. And it was leaving them and into the air, both directly and from the bike.
While there is a bit of heat loss through the metal parts of the bicycle, I think your experiencing more of a physiological response to the cold than a thermodynamic one. Feet and hands get cold due to decreases in circulation as the body pulls more heat to the core. The surface area to volume ratio is higher for your fingers and toes so you radiate more heat there as well. Add in shoes that fit tighter and gloves that fit tighter and everything starts to works against you keeping warm.