Moment is the critical factor and as most of the weight is under my bum it has minimal effect...also we are talking about pretty low gear weights.
Not sure how true the first part is or isn't, but I think the bolded part is kind of key here and the lighter the weight the more so. The more weight you carry there the less true it will be.
I will say that I have toured with a little lighter load in a similar location and found the weight's effect on handling pretty minimal.