I have just one pannier for about 8 months out of the year. It usually weighs close to 20 lbs. I've had no issues.

I really don't even look (or care) to see what I'm hauling. My tubus rack is rated for around 100 lbs and I commute with an ebike. In the winter I add another pannier full of clothing (layers, extra socks and extra gloves.)