Wind is the deal breaker.
I have ridden with temps around -5° C before, but sunny and almost no wind, I was fine. It only happens on very rare occasions during the winter in my area.