Not enough to matter. Maybe if you were carrying bricks or lead bars or something.
I use a grocery bag pannier on just one side... I've carried a couple gallons of milk and OJ and stuff so a pretty decent load and the fact that it's off-center doesn't affect you too much. Think of it this way, I dunno what you weigh but I'm 200 lbs. A 20-lb off-center weight isn't going to affect my overall balance on the bike much.
__________________
Treasurer, HHCMF Club
Now living in the land of the cheesesteak.
But working at a job where I can't surf BikeForums all day any more...