I Put something over my sweater, The knit is not much of a wind block ..
I see business men riding in suits. How can they do that without showing up to work a sweaty mess?
Start early , ride at a dignified Pace.
In the rain you could a try a rain cape.
this.. cycle rain capes are cone shaped .. Ponchos are flat.