Challenge is one thing. Strain is something else. Damage, beyond that.
Can sometimes be hard to know which is which, with a given activity. One nice thing about conditioning and fitness, though, is that (in general) it helps raise one's threshold and guards against such threats better than if one is less fit.
Back in the day, I used to run distances. Often quite hard. But in the colder times of the year, it'd be with appropriate clothing only, and it'd be at a pace and intensity more-suited to the situation. While at the time I'd never heard of cold weather being any more of a threat to heart, lungs or anything else, it made sense that the added stresses cold could impose would, at some point, be an unwarranted risk. And so I'd typically ratchet-down the intensity on such days. If for no other reason than avoiding the icy-cold part of the impact on the lungs, cardio-heavy that running is. Nicely, such overall fitness stood me in good stead during back-country hiking, ocean swimming and a host of other outdoor activities in less-than-hospitable conditions. I doubt I'd have survived half of the bad days, on such excursions, had I not been that fit and trained in anticipation of such days.
Pros and cons, as with anything we do.