Strange. I was just pondering the same thing this morning.
The increased weight has quite a bit to do with it IMHO. Plus, I find cycling in cold weather much more physically demanding. Cold air seems to make my lungs work harder and the humidity is slim to nonexistant.
Just think when spring hits... much better shape for riding with a jersey and shorts.