That's exactly it.
I have a cheap rack on one of my bikes, and a Tubus on the other. The whole cheap rack twists and sways with a load. The Tubus is like a bridge. I can easily forget the panniers are there.
Also, some panniers have elastic shock cord at the bottom (rather than a rigid toggle), which can allow lateral movement with full panniers. I've had this happen - the panniers slap the rack.
Either situation can be rectified, whether it is the rack itself or the pannier attachment.