I generally only start thinking about wearing something on my head (other than my helmet) below 35 or so. I've ridden down to about -25, at which point I'm wearing a gaiter, a balaclava and a neoprene face mask. In between, I just put on what seems like a good idea at the time.

Even if I start out quite cold, within about 5 minutes I'm making enough heat that it's not a big deal. My ears need to be covered. I was riding a few days ago at 15*F with a stretch gaiter up over my head and ears but with my face completely uncovered. It was fine after 5 or 10 minutes, a little too cold before that.