You have more than one pannier? For commuting? 5 years..one Arkel Utility Basket...no problem.

I use folding pannier baskets. The left side is always open with a bag in it.
When I first got them, for whatever reason I would put heavy things on one side and lighter things on the other.
It didn't take me long to find out that, that hinder the way I rode because the weight kept easing me in the direction of the weight.
Now I just split it.