I find that much of any weight up high really affects off-road handling when the going gets tough, whether it's in the handlebar bag or on top of the front rack. Once I moved most of the weight down into front panniers the bike handled much better. This is especially noticeable for me since I use trekking bars and with the hands close together while braking, etc. I don't have much turning leverage. I took one off-road trip of 35 miles where that combination of high weight and low leverage put me on the ground multiple times (ouch).

The high weight doesn't seem to be that much of a problem on the road, you get used to it. I still use a handlebar bag but I'm careful not to put too much weight in it.

Bruce