For some reason my intuition is telling me to try moving the panniers forward, keep the heavy stuff at the bottom, lighten the h/b bag, and moving a little more of the total load from the front to the rear panniers. I really have no data to back this up, though.
Also, it could just be head-tube angle, i had a super twitchy mountain bike with a steep head tube.
You might get used to it, if you do, that's great! Give it a few hours of trying. But, from my own experience, if you have a handling problem that doesn't start to feel OK in a few hours of riding, it is worth it to sort it out (including bike purchase if needed) b/c crashes really suck.
If you ever want a local short-local-tour partner, or if you tour through here, give me a buzz, I'm in Boulder.