Something is clearly letting freezing air inside. As many pointed out, balancing front wind protection with breathability is a common problem with cycling in cold weather and it works differently for everyone. A vest or jacket designed for cold weather will be often referred to as a "barrier" vest or jacket. The entire front is windproof and waterproof (including a good zipper!) and the sides and back are breathable. Not every cycling vest or jacket is designed like that.