The coldest I have ridden is about 10°F, but I've found that a helmet cover and a balaclava/beanie are sufficient and much more versatile.

If you have a cold morning commute and a mild afternoon ride home, you're going to be hating the ski helmet.

You do make a very good point about cold weather polymer toughness. The colder it is, the worse the problem could be. Ask any engineer from the Space Shuttle Challenger days.