You need a gradation of clothing for colder conditions.
You dont need a seperate base layer with a jersey, most are polyester wicking fabrics. Use a base layer wicking T shirt if you need the insulation. Sometimes 2 layers of jersey is just the right amount of wind proofing.
Get a non waterproof, ultralight windpoof jacket or smock. They are eay to carry and very effective. Maintaine featherlight is a nice example.
I have never ridden in conditions cold enough for a balaclava, this would be below -10C. A neckwarmer tube is probably more versatile.