Although I won't start a ride in heavy rain, getting 'rained on' doesn't bother me particularly, nor does cold. I'm quite happy to ride in temps down to around -5C -- even a little below.
What does stop me, though, is snow/ice. Once the roads round here are iced-up, I'm off the bike for the winter. This year, starts today: currently -12C, and we've about 30cms of fresh snow on the ground.
At age 60 I've no desire to do a slip/fall and break a hip or something; I also have no desire to watch out for our drivers, who are even worse in the winter than at other times of year. Typical: texting/on the phone, no snow tires, sipping coffee, oblivious to everything (including traffic signals and her/his incompetence).