I've put more than 58,000km on a Fuji Touring doing randonnees, touring, commuting, training and leisure riding. It started out with 32mm 700C tyres, and over a period I put narrower wheels and 23mm tyres on it to improve its speed performance. I even toured on that tyre-wheel combination, but I did wreck a couple of rims doing so.

Maybe invest in a second pair of narrower wheels and tyres with the same cassette set-up so you can just swap them with the originals.

Having said all that, touring frames such as the Fuji Touring, the LHT and Trek 520 are designed to be loaded and to ride and handle best that way.

You might also consider reducing the pressure in your tyres compared with what you run when loaded. I usually run between 90 and 95 psi with all my single bikes, and maybe up to 100 if I have a load on board. The lower pressure improves ride and makes the bike a little less twitchy over bumps in corners.