Imbalanced weight is unconsciously compensated for by tilting the bike slightly so the center of gravity is over the line between the contact patches. We don't think about this, we just do it, and unless the needed tilt is severe will not notice. Keep in mind that most of us spend most of our miles riding across the slope of cambered roads. (check your tires and note that they are worn slightly off center.

So, go ahead and ride with the single pannier holding whatever you need to carry. 2,000 miles from now you might be replacing your tires a little bit sooner than otherwise (maybe).