I think a pannier is close enough to the centerline of the bike that it would be very difficult to overload one enough to unbalance the bike while riding. I know I ride with one on the rear all the time and don't even notice it.

I would hesitate a bit more about riding with just one on the front, since I can feel the difference in handling with just a couple of pounds more in one front pannier than the other.

RichC