You were wearing what I would consider appropriate clothes in pretty mild temperatures. I wear a polypro undershirt, a lycra jersey and a thin sweater or fleece under a thin jacket in very cold (-15 C or so) weather and I don't get cold... wel, I get a little cold at first but after 5 or 10 minutes I warm up nicely.

The problem sounds like cold air is sneaking in around the collar of your jacket... If I do not zip my jacket all the way up to my chin (even though the zipper snags my neck whiskers) the top of my chest gets cold. Try using a neck tube or Buff to 'seal up' the top of your jacket.