I have to fit a decent-sized laptop in my panniers.
Hence I bough a set of large panniers. If I've got few things to carry, I only take one.
It's nice to be able to fit 5 days worth of clothes in the other Pannier though so I can make the bi-weekly clothing run without needing the car (I usually wear each outfit twice, btw - it doesn't get too dirty sitting at a desk).

+1 to whoever said shoes take up a ridiculous amount of space.

I can't see what I'm losing out on by having more storage capacity than I use.