i carry two panniers every time i ride. lunch in one and book/ipod/ badge/ wallet stuff in the other. when it is warm in the evenings it is nice to have the space to carry some of the clothes that kept me comfortable on the way.
i dont think I have ridden to work with one pannier ever.