My preference is panniers. I can and have done backpack hauling for years, both on the bike and off. And up to 20lbs or so, a backpack is no big deal. It does make the bike a little twitchier, but that's my fault, not the bike or the pack's (I'm clumsy, so my balance isn't always as good as I'd like). I just suck at managing the weight right . I very often have loads in the 40lb range, and 60lbs is not unheard of... and at that point a backpack is sheer torture. Sweaty back isn't the issue at that point, it's shoulder bruising... and no amount of dropping the pack down will make it weigh less.

Panniers make managing the balance easy, even on a short wheelbase bike. And on my cargo hauler, even 55-60lbs is doable. The main trick is to keep the load balanced and sensibly sized for your bike. My last bike could only manage a rear load and had short chainstays, so it got rather scary to ride with a 45lb load. Some bikes have an even shorter wheelbase, and might cap out at even less weight.

Right now, my panniers don't work very well if I'll have a lot of time off the bike. If I know that's part of my day, I'll use my backpack. When my cheap panniers wear out, I'll look at getting more flexible ones.