I load the non-drive side first for reasons already given and one other; I have two panniers with holes in them and in both cases it's from rubbing a guard rail (my main commute has a narrow section of shoulder between a guard rail and a rumble strip.)