Then don't rock your bike. Learn how to stand and pedal without throwing the bike all around.
You want fun? Try two panniers loaded with groceries (about 30 pounds each) and standing. If you rock the bike then, the pendulum effect can knock the thing right off its wheels.
In other words, the single pannier isn't the problem. IME.
I learned that it was more energy efficient to rock your bike when riding out of the saddle as opposed to rocking your torso and keeping your bike straight, which is why I opt for the former.