I have two panniers (Ortlieb Backrollers). I usually ride with just one, though, and even when loaded down, I don't notice after the second turn of the cranks.

What Juha said is true, though -- groceries eat up a lot of space. You may need two. That's when my second pannier comes into play -- grocery runs. Even then, I'm often strapping something big (like a dozen rolls of TP) on the top of the rack.