I hated the way my Surly LHT handled with front and rear panniers. Steering was unpleasant for me. I took off the front rack and bags and now tour with just the rear panniers and a very light dry bag on the rack in back. I know I'm overloaded in the back, but I have not felt that the handling is compromised in any way. Getting that weight off the front of the bike has mde riding a joy again.
I do keep a medium/large handlebar bag for big camera and sundries.