Try some vasoline on your face. I put it on my forehead, cheeks, and nose. It helps.

I agree with others who suggest you are overheating. It's hard to do, but try to dress so that you are cold for the first five minutes, but warm then after. If you bike hard, the neck and face areas get warm quickly. That turns to sweat, which if you're dressed too warm will build up and give you the chills.

Everyone's different, but at -5 C, I wear just one thin bacalava under my helmet. Try ditching the bandana and put on vasoline, see what happens for a hard 10 minute ride with your two bacalavas. Then keep on iterating with stuff until you figure it out.