This ^^

I've tried riding with 4 panniers, but just couldn't come up with enough stuff to justify it.

I also try to ensure that the weight of my bicycle + all my stuff comes in at approx. half my body weight ... or less if I can manage that. So even though I usually carry most of my stuff on the rear rack, it isn't that much stuff.