I want to add that riding in colder weather is mostly about the right gear. At colder temps, gear that's windproof but breathable is the key to both comfort and avoiding permanent injury from frostbite.

I could not believe how much more comfortable I become after buying truly windproof gear.

There's 2 things I personally recommend -
1. Gore Windstopper stuff if you never ride when it might rain.
2. Shower's Pass waterproof breathable gear if you ride when it might rain. I can't believe how windproof this stuff is. The one caveat is that it's not breathable enough at warmer temps - like, above 50 degrees. To be fair, nothing's better that I was able to find. It was good for me below those temps. Some people who naturally sweat a lot felt it wasn't breathable enough, but it worked amazingly fantastic for me in the last month or so of winter last year - it was one of those things where I didn't even know how cold it was out until I tried biking with it unzipped.

However, above the actual freezing point, you can also get away with more layer and normal biking gear just fine.