My touring on and off pavement has been with front panniers only and a dry bag stuffer on top of the rear rack for the last 25 years. Many of the advantages have already been stated including ease of pushing with this set-up. I have never found crosswinds to be a big problem. I was only tempted to move the panniers to the rear rack once during sustained 40+MPH headwinds.

My front panniers are mounted high on a Bruce Gordon Mtn. Rack which places the panniers partially behind the steering axis which I find very effective in stabilizing steering even on single track and trails. I too keep the equipment weight minimal but functional. I have been quite happy with the arrangement and have seen nothing that would make ne want to change now.