Maybe something like this in black when on sale since it is quite expensive if not (smartwool PHd HyFi divide half zip)
This one (the yellow part is made of wool) is to show you the wind resistant membrane in front (nylon). It will probably replace your jersey rather than go under since it is maybe a little too hot for 2 layers in the 40's






Another one lighter and way cheaper with a wind resistant front panel

SF-2 plastron from Louis Garneau


My advice is to have a wind resistant/proof membrane outside to cut wind before the cold wind reaches your chest and still have the insulation layer to insulate from cold

A vest with a wind resistant front panel like the nova vest from Louis garneau