I see no issue with a ski helmet in cold weather.
Keep in mind, performance fleece can be surprisingly thin and warm. I've ridden with a thin fleece cap under my helmet and a shower cap over my helmet in freezing rain with no issues (and I HATE HATE HATE the cold). A heavier fleece neck gator and balaclava really help, too.