If you are bigger I would imagine that the added touring weight is killing your partner's weight advantage. He is used to tapping up hills and now had to drag an anchor behind him. As a heavier rider, you have been carrying that anchor the whole time. Even if you guys carry the same load, his power to weight ratio is hurting more.