I find a lightweight windblock layer to be really important. If I have something reasonably insulative on, and then can keep cold air from whistling through it to my skin, I'm usually pretty comfortable. The particular item I use is the Rapha gilet--it's lightweight enough I can wear it in summer and not overheat, but it makes any winter outfit immensely warmer. It's probably my favorite/most used item of bike apparel.

I'll echo SeattleForrest to advocate for layers and wool. Wool keeps better insulative properties when wet than most fabrics, so if you've broken a sweat up the big climb, it'll still feel warm on the way back down. (Or if you're out in the rain, etc.) A hat also helps me a lot. I have a thin wool cap that packs up smaller than a Gu packet, but makes an amazing difference when I put it on. Shoe covers and serious gloves also really help, though for me, they are only effective if I get my core temperature up. If I don't get my heart really pumping, nothing in the world keeps my fingers and toes warm.