I will add to my original post... while for some it may be imperceptible, you'll notice usually if you follow someone with one pannier they usually have a noticeable list to the side that the pannier is not on.

I'm sure it's a balance issue, but for this old man (crappy left knee) that tilt throws my geometry off enough that in the long haul it causes me pain. To each their own tho.