I have a cheap wicking tee to go under my long sleeve jersey.

I use the newspaper wind vest trick if it's a bit too cold at the start of the ride, but expected to warm up later. Take a sheet of newspaper and put it under the jersey in front. It blocks the wind chill, and you can throw it out when done. It works for me. I read that riders do this after they've climbed a mountain road, and it's going to be cold on the way down.