The difference with cycling vs other winter activities is that you're getting hit by wind all the while you're riding. I've had that cold torso problem and it was usually because I wore too much then unzipped my jacket and got wind-chilled in the front, while the rest of me was still toasty.
I wear a light sweater and a wind-resistant jacket that has a light fleece lining. I'm fine as long as it's zipped up and the cinch cord isn't completely loose.