Using only a single pannier causes you to be visually unbalanced, which of course as we all know can cause a significant speed loss. Using two panniers will balance the aesthetic load and result in a dramatic increase in speed.

On a more serious note using a backpack will raise the center of gravity while panniers lower the center of gravity.