I commute always with only one pannier. However saying I only need one is not completely true. I am still saving up for the second one...
I believe I can feel it a little bit if I really load it up and ride free-handed.