Assuming you mean plain polartec, adding a nylon shell will keep the wind from taking away the heat trapped in the fleece. Also, if you gloves are roomy enough, try a pair of thin polypropylene gloves as inner layer. If you have bar ends, cover them with tape or grips.