Most GOOD panniers have straps to help pull in any extra material that you don't fill. Although my usual load is the clothes I'm wearing at work, I went with Arkel panniers for touring (can do double duty.) When I have light days, the pannier compacts for me. I can still get up to 25mph with one pack on the left side of my bike. When I need more room, I have it.
I find that using one pannier, with a varied load, doesn't affect my biking at all.
I would say that you should get a good quality pannier, with compression straps, that can handle your maximum load (or what may be your max load.) Stick with the major players and you'll get a good pack.