I started off with one pannier but couldnt decide which side i like so I bought another pannier and have no such problem
But when I did have one pannier, I started off on the drive side figuring if the bike falls down then the pannier will protect the rd. But then I put it on the non-drive side because I figured the pannier was heavy and if I fell down, it's more likely I'd fall on the heavier pannier side which was safer because it's away from passing traffic (UK, we drive on the left). In the end, that one pannier was too heavy so that's the real reason I bought a second one.