Experiment with the pannier placement. For me, moving the panniers, the rack actually, more forward improved the handling. Like on the above irwin7638's picture, my front panniers are now in the middle of the front wheel.
Also consider air drag at higher speeds: if your panniers have compression straps, use them. This will also ensure the contents of the panniers won't move around which would definitely cause poor handling.