Back when my wife and I were both physics grad students we regularly commuted to campus on our tandem with only a single pannier. Never noticed any difficulty as a result of the unbalanced load, nor did anyone else in the department ever express any concerns about it to us.

Certainly having the extra weight on one side meant that the bike was leaning very slightly to the other side so that the center of gravity was kept centered over the line between the tire contact patches. But I don't see any harm in that nor was it perceptible to us while riding. And I still sometimes go grocery shopping with only one pannier and may have it loaded with up to 20 or 30 lbs. depending on what I'm buying. I do notice the slight lean with a very heavy pannier on one side of a single bike, but it's still not a problem and the bike handles fine.