You wore more than me (upper body) on my single digit morning commute today. I do, however, wear compression pants over chamois, and a pair of breathable pants to block the wind. I personally wear goggles and a balaclava if it gets below 20 degrees or so, as the wind on the eyes (I wear contacts) causes my eyes to water excessively. I also like mittens instead of gloves in sub freezing temps. And I wear boots over wool socks, and ride platform pedals.
One thing I will say, is that when the temp gets down near 0, any kind of headwind will penetrate and cool unprotected appendages, including the one between your legs. I carry an stocking cap to stuff down there if needed...no fun arriving to your destination in good order, but with a frozen banana.