If you have not noticed, your heart rate jacks up much higher than normal in the cold. It is amazing how hard your body works to keep warm.
Same in the hot summer with your body working harder to keep cool.
You can train around this to some degree and improve a little.
My MD co-workers say below 20, there is not much exercise benefit from riding.