You're probably wearing too much. Take off a layer and you might be more comfortable. Syntheticc fabrics and wool are usually best. Try a wicking Tshirt, a polar fleece hoody and a light wind-resistant outer shell. On the legs, you can try bike shorts (if you like them), polyester longjohns, and windproof hiking pants or wind pants. Get good gloves, socks and hat.

Combinations like this have been working for me for the last 6 years of winter commuting.

Good Luck.