I agree with you. I've ridden in temperatures down to the single digits °F (-10 to -20°C), and I've skied (and sledded, walked to school, delivered papers...) in weather down to the -30s °F, but the coldest I've ever been was during a ride where I got caught in heavy rain with temperatures in the low 40's °F without a wind barrier layer. Like you, I finally made it home and got into bed with every blanket we owned, shivering violently for over half an hour.