I don't mind riding with one loaded rear pannier, but I hate getting out of the saddle with weight back there. My preference is to put serious weight in front Low Rider panniers, even though I have to use two panniers because asymmetry is less than safe riding one or no handed. Hence half my bikes with racks don't even have rear ones.
The Ortleib panniers (front) that I use have good handles so carrying both in one hand is easy.
Ben