Roll the jacket and put it in the pannier. Wear a short sleeve dress shirt. I arrive in fine shape after a 10 mile commute. One of the advantages of cycling is that the relative wind makes it far cooler than walking.