I doubt that pannier weight matters little. All racks have weight ratings and I own quite a variety of racks - there is a clear difference between the ones rated for really heavy loads and more typical touring racks.
Sorry. That isn't quite what I meant. I was only talking about what the bike can support.