Wear a short sleeve shirt. Leave your sports jacket on the rack or in the pannier. Don't to push yourself too hard. Remember, a level of exertion comparable to walking gets you going about 12 mph -- that means lots of cooling air.
Mjw16 has a really good point. When you come to a stop, your metabolism is still elevated, but the cooling airflow has stopped. That's why you want to get into the a/c as quickly as possible -- even if that means walking in the door with your helmet still on.
Done properly, you shouldn't be be in any worse shape than the person who walks a few blocks from the subway.
Paul