Give it up -- were you trying to ride shirtless? No, I have a "non-traditional" opinion of over-priced bike-specific clothing and am not the least surprised that a thin plasticky vest didn't work for you. Others have mentioned trying wool, but my personal choice is regular LS shirts and a good polartec (polarfleece) vest. Those really cut the wind and work great for me -- easy to move in too.

Gotta keep your core warm! You're not a shaved head guy like me, so you're probably good to go there .