What about buying one of those Buff face masks fishermen and hunters like so much? Looking at their website they have even started to market a bit to cyclists. It is supposed to keep the wind off your face but also be pretty breathable. They might look a bit dorky/******y, but if it works, I could live with that.
Buff®Sports | Cycling | Original