45 miles in 25 degrees for the new year, wearing jeans and a light zip cotton hoodie (started with a light down vest but got hot and took it off). Felt great except for a bit of sweating/freezing/almost-chafing in the saddle area near the end.
For me one of the biggest benefits of switching to fixed was the feeling that, no matter what I'm otherwise wearing, I'm ready to ride. Maybe even more than the fact that I just love riding my steel fixed gear compared to my carbon geared bike, it's the regular clothes aesthetic that's been responsible for getting me far more miles and far more enjoyment out of riding since making the switch.
It's all psychological, of course--no one forces you to gear up when you ride a geared bike (and I was always pretty minimalist anyway)--but for me the compatibility of my bike style with my personal style makes all the difference. YMMV, and if you dig the comfort or aesthetic of cycling-specific clothes, that's chill too.