For temps into the 30s I wear a base layer and a breathable winter jersey and over that I wear a Windstopper vest. Low 30s I might add a heavy jersey and maybe a jacket. I always wear bike shorts but when it's cold I add leg warmers. I have some that are light and some heavy.
Long pants with chamois are available, as are windproof cycling pants. You can wear long johns or tights.
50s or lower I like a wind resistant cap that covers my ears, it's a big help.
Most of the time I wear glove liners under fingerless gloves. If it's really cold I have neoprene/leather gloves or when descending in the snow on the mtb I wear ski gloves.
I like Merino wool socks and neoprene toe covers. I've been in the 20s and used chemical warmers under the toe covers. It works great and I learned it from a BF member.

I choose a lot of stuff based on the fact that I sweat buckets. Even gloves, if they don't breathe they get soaked. It's why I prefer a vest over a jacket most of the time, I can take it off and stash it for a climb.