So far this year I've been missing winter riding because this past February and March I was riding with ski goggles and a ski mask for some of the very same reasons that the OP cited, save one: I can't take the heat and sweat like a stuck pig. Sweat literally rolls down my brow and into my eyes. This was a non issue in the winter because A: I don't sweat as much in the winter, and B: the padding around the goggles kept any trickle from going into my eyes.
Sure, I got a lot of laughs from joggers on the local MUP but I found winter riding much more enjoyable this year than I did even during the heat of summer.