A good base is the foundation of keeping you comfortable. For cold weather riding, don't pay the roadie tax with cycling-specific garments - I wear Smartwool t-shirts that I've found on closeout in the past.

Leg and arm warmers are great for rides with more variable conditions, so you've got a good start. Likewise, vests are really versatile and highly recommend; a vest over base and LS jersey would work for weather like this, too, and then you could unzip or remove the vest as the day warms.