I think you are wearing at least one layer too much. I always start out cold and expect to be comfortable at the halfway point. At warmer temperatures (30s), I'll often arrive at work with my coat completely unbuttoned and gloves in my pockets.

Paul