I wear shorts, a t-shirt, polar fleece hooded sweatshirt and a hooded rain jacket. None of it is bike specific, just stuff I had around the house. With both hoods up (under my helmet) I'm not quite warm enough when I start out. I just don't get a workout until after the halfway point. Once I hit the hills it's a different story, I warm up quickly and no longer feel the cold.

My wife had to point out the obvious to me, I can pull over and adjust my clothing if I need to. (after decades of car commuting that is an alien concept) I've been trying this, pulling over after the first hill and taking off the hoods, and that seems to make enough of a difference that I'm not overheated by the time I get to work.