I start wearing a very thin balaclava (just stretchy woven polyester, really) around 30F, and it has worked fine down to around 10-12F.

At 20F, I might start being concerned with frostbite if I were going very far without ear covering. It depends on how long you're going to be out and how fast the effective wind speed while riding is.