I'd highly recommend some type of "technical" undergarment. Cotton tends to keep the moisture trapped against the skin and can get leave you chilled. The synthetic stuff wicks the moisture away and works super. Often it is just the additional amount of layering needed when a single jersey just isn't quite enough. Velowear has some fairly reasonable upper undergarment stuff-especially on sale. It is perfect for what you described.