I've been commuting with one pannier for 25 years, with varying weights and no issues. It's an easy experiment to try. Just load up your pannier and see how it feels. You'll get the answer for yourself a lot faster than asking a bunch of strangers.