I think any improvements will result from moving the bags back if possible. I agree with B-Gordon that the weight should be centred along the steering axis. After this consideration comes lowering the weight. All of your weight is in front of your steering axis (in the picture).

I use a tubus duo and I can easily carry 20-40 pounds total without ANY effect on handling provided I mount the panniers as far back as I can. If I mount them forward a bit on the rack, i start to notice 'flop'. Also, Riding hands-free down to a speed low enough that my dynamo stops working is possible (~3-5mph).