I've towed only a little bit, but vastly prefer the way a loaded bike handles. Weight on the front adds weight overall (in the rack and bags,) but getting the bike balanced front to rear takes weight off the rear tire and makes for a more stable ride. Also, having a little excess carrying capacity makes packing after a shopping trip easier.

Another reason I'd recommend a front rack system is that if you decide to do a super lightweight trip later on, you'd have the option of leaving the rear panniers home and using the front panniers for gear with a bedroll on the back, which is my preferred way of packing. Here again, I think a bike just handles better packed this way. I've had front panniers for years and only bought rear ones last year.

I use a Salsa lowrider on my Vaya. For the rear rack I've had a Blackburn Mtn for years. Somehow it's survived a number of gravel road tours.

Carrying too much won't kill ya. Have fun!