You have discovered the secret of bikes purpose-built for loaded touring. Unladen, the ride can be anywhere from meh to pretty nice. But they come into their own under a properly-mounted load. It is amazing how it can feel like you aren't carrying a load at all (except for when you are starting up, or going uphill, or trying lean the bike against something, or getting on or off). But at speed on flat or gently rolling terrain, or on a descent, it feels great.

BTW, I'd be careful about setting up the second rear brake with a shift lever. On tandems. it was common practice to have an additional rear brake (almost always an Arai drum brake) at the back running off a shift lever so you could set it as a drag brake to keep your speed under control on long and/or steep descents. Having captained a tandem that weighed about 500lbs all in (250lb captain, 200+lb stoker, 45+lb tandem and some additional gear), I was always glad to have that drag brake. It didn't so much slow us down as make us accelerate less quickly. Without it, when you let off the brakes on a significant descent, that thing leapt forward like it was shot out of a gun. I'd be leery of doing the same thing with a caliper brake because of heat build-up on the rear rim. Having that kind of heat build-up on the rim does not guarantee blow-outs happening, but it certainly increases the chances. With the drum brake or, more likely nowadays, disc brake, the heat is far, far away from rims and tires, so no potential heat build-up issues.

Someone will no doubt chime in to say if tubes and tires are properly mounted you shouldn't have an issue, and maybe that's so. But what happens if the tube/tire isn't mounted perfectly and you don't notice it? Color me conservative, or wussy, or whatever adjective you wish, but however small the risk, it is easily avoidable and I'd rather not take it.