The steel SPD cleat, clipped into the steel and aluminum pedal, screwed into the aluminum crank arm, with a whole steel bike frame at the other end of it, is an enormous heat sink. In warm weather that's not an issue, but on recent rides I've been getting really cold feet despite pretty good insulation.
How cold was it? I have seen that be a problem but not until it is pretty cold, like maybe single digits F or lower. The coldest I have seen on tour was probably 18 F but even then it warmed up pretty quickly once the sun was up, so for me the heat sink thing has never been an issue on tour.