By the sounds of it, I'm fortunate to have a rack and panniers setup for my 4 mile commute. Also, I keep a frame mounted pump as well as a tube, tools, levers and a pair of rubber gloves stashed conveniently in my saddle bag. The logic being if its raining or just very cold I can save my gloves by chucking those in my jacket and using the rubber gloves to keep me and my gloves clean and or dry. Being that they're in a saddlebag access is always immediate instead of emptying my pannier.

in one of my panniers I always keep a u lock, a very lightweight wind layer, a rain jacket, beanie, a merino wool buff, and toe covers. When it's really cold, I'll throw in a sweater and use it for layers. I know it sounds like a lot but, rolled up in the bottom of a pannier that is only used to hold my lunch, it's really not cumbersome.