I worry about the cold air hitting my chest, that's why I use the waterproof jacket or the t-shirt with the teflon layer on the front.
Today I used the waterproof with a polartec t-shirt underneath to keep sweat away, and it worked nicely. Took the shirt and a pullover in my briefcase pannier and put them on when at work.
Casual looking clothes are a plus for me since I'm a teacher and a real expert in getting to my school just in time for classes.

Is there any windproof ventilated jacket you recommend?