Proper clothing is for sure critical. I have a ton of experience with cool to freezing temp riding as I had no car for nearly 20 years here in the frozen tundra of the north. Below zero temps and a blizzard was unable to stop me.
Today I am retired and have a car. I don't need to relive my youth! lol
This is why I can't acclimate to the cool temps: Burn out. Did it and can't to do it again.