My old commute had a mile or two of single track that I would ride with my lightly loaded (clothes/lunch/work boots) panniers and I never had any issues with them and there were plenty of rocky/rooty sections of trail. They are even cheapy panniers from Axiom.