35°F is my cutoff point. I wear a thin one, and even that I have to pull down when I stop. My ears, however, need to be covered below 50°F or else they end up feeling like they've been punched after a few miles.