Well, I wore a balaclava this morning when it was only 4*C, but I used a very thin one that covers my ears and neck.
PM ride home was without, of course.
I'll use the same balaclava down to about -5 or so, but with it covering part of my face. Colder than -5 I'll got to a thicker one.