What kind of jacket do you have and how cold it was? A cycling jacket that has wind/rain barrier on the front but breathable sides and back can make huge difference: it'll keep you warm an dry. The key is to have a protection against the cold wind but enough breathability to push the heat and moisture out.

I use long sleeve wool base layer, then a wool sweater (I have two: thin and thick with turtleneck for diff temps) as second layer and a wind-barrier cycling jacket or light general purpose winter jacket depending on the weather. I too avoid cotton, it's horrible: keeps you cold in Winter and hot in Summer.

My cycling jacket has wind barrier on the front and it's breathable on the back and sides and it works well with a thin wool sweater down to around 32F. If lower than that then I need to switch to a winter jacket that doesn't have large ventilated areas but it's still breathable underarms so I can push the hot air out while moving. I picked up two cheap 100% merino wool sweaters from GAP outlet on sale, they have some imperfections but they work great for biking and hiking. For $30 each they're hard to beat. One is thin and the other is thick. But for the base layer it's really good to have something high quality made for cycling, it makes a big difference indeed.

Adam