Although I always bring a backpack on tour I do not wear it while riding for many of the reasons mentioned above: Hugh center of gravity, overheating and sweating and saddle discomfort from extra weight on sit muscles and bones.

In the mid-80's I toured the Andes of Ecuador and Peru for several months using a backpack mounted upright on the rear rack and two front panniers. That arrangement worked well for a dirt tracks and rock strewn routes on volcanoes above 4000 to 5000 meters. When pushing the bike was required I could wear the pack while pushing the now lighter bike and no rear panniers to get in the way of my legs.

I still tour on and off pavement with a similar setup with a dry bag stuffer on the rear rack instead of the backpack. The backpack provides versatility loading up groceries to carry to the campsite and of course, for use while day hiking when biking is prohibited as in much of US national parks.