In summer, take your jacket off, roll it (do not fold) and stash it on your rack or in your panniers. Wear short sleeve shirts. If you are in a casual enough environment to wear a sports jacket, just leave one at work and rotate every so often.

Paul