I hope this helps--today I rode in with my grocery pannier to work, as I needed to bring in some coffee supplies. I had a 5 pound bag of sugar and a large can of Folger's coffee, plus a change of clothes and my lunch. It rode fine and I got a little extra excersize!

Don't sweat it!