I can completely understand a fear of the cold. I've already slowed way down on my cycling because I watch the temp and keep saying, 'No way'. For me the problem isn't during the ride. I dress as recommended and what feels right, feel fine for the ride itself. Just when I get home, about 10 min later, I start shivering and even with hot baths and drinks, I'll shiver for up to 4 or 5 hours after the ride buried under a mound of blankets.
Definitely puts me off cycling at this time of year.