The worst part of pushing a loaded bike up a climb is hitting the inside leg on a pannier. I have often given thought to removing the offending pannier and strapping it down to the opposite side of the bike so that I can be more vertical when pushing instead of having to lean over from one side when pushing.
Have you weighed your packs and gear?