Sorry, no science to back me up. All I have is my own personal experience. I usually only have one pannier on my bike. I don't even notice it unless there is a huge load in there, (i.e. a gallon of milk or a bunch of canned goods).

The differential between the two sides becomes more noticeable at lower speeds and I usually try to ride fairly quickly.