Cyccocommute: If you hold your shoulders perfectly still relative to your bicycle as you ride, then I will concede that all you are losing is the advantage of a much lower center of gravity for what is probably the heaviest single thing you're carrying on a tour.
But I bet you don't! Ride alongside another rider whose style and fluidity you admire and watch their upper body. Motionlessness you will not see.