Depends on the weight and what kind of rack you have.

I try to keep the weight below 10KG in my panniers.

I've noticed my (cheap, fairly weak) rack bending to one side, when only using one pannier.

I use 2 panniers to balance the weight better if I'm going to be carrying any significant weight.