I use both panniers. I have ridden with only one a couple of times. I like both for a couple of reasons.
1. The symmetry. I don't wear a watch because nothing is on the other arm.
2. I don't like to cram stuff in one pannier if I don't need to. I would rather use two. It is easier to get what I need.
3. Probably the most important. I don't know what I might do on the way home or buy at work. I like having the capability to carry more. There have been times when I only use a trunk bag or one pannier and I needed the other pannier to get what I wanted.