If I have two panniers with me, I'll distribute the load more or less if I can. Seems commonsensical.

But I don't notice a difference if I don't.

Maybe I would if rode really aggressively, but I don't do that. So for just riding around the city and on paths, I don't notice a difference if I just use one.

I dont think it makes much difference, but if it did, you'd probably only notice it in fast curves. So if you don't do that, you're probably ok.