Get a jacket. But for me at least the key to cold riding has been Under Armour. It's made it possible for me to ride comfortably where in the past I just wouldn't go out.
That said, i have an old Plumbline with a Gore front that's amazing, but usually too warm for California.
Usually my base, my jersey and a Performance Illuminite is enough.