I've also often commuted or shopped with only a single pannier. If I buy some heavy groceries and load up with 30 lbs or more on one side then I do notice a slight lean, but it's never been objectionable or caused any handling issue.

This guy's load is a little extreme with a refrigerator on one side. He is leaning a bit, but doesn't seem to have any problems with it.