What is the t-shirt made of? Cotton? Cotton is bad, it gets wet and stays wet & cold.
It's not rocket science, and everyone is different as to what to wear in colder temps. You will just have to try out various combinations. Sometimes 2 pairs of armwarmers, 2 jerseys, vest, sometimes long sleeve jersey, wind jacket or warmer jacket. You'll definitely need some long tights for the 40's. Wool socks are really nice, too.
It's a big investment in clothes, but try to get items that are different weights/types and you can layer them to tune them for the temperature. Try to get jerseys and vests with big pockets to use to carry layers as you shed them.
Warm gloves, toe or shoe covers and something to cover your ears can help a lot.