Depends on the bike. Panniers affect the handling less if they are weighted low and toward the center of the bike, and if I put a rack on my road bike the weight of the pannier will center behind the rear hub and the bike's handling gets wonky and not in a fun way. Backpack is the way to go there. Within the limitation that it has to be cycling-friendly, ie no internal frame and short enough to not impinge on my neck or butt. So I would say your huge capacity perceived benefit is a bit optimistic. With regard to your concerns:

a) sweaty back is trivial, and where I live it's not hot so it's only ever insulation.
b) I have a wonky back and backpack while riding is never an issue (riding in general is never an issue, it's only off the bike that I have problems)
c) since the capacity is limited, there is no real problem w/ me carrying the weight instead of the bike and for the roadbikes in particular it's better to have the weight on me than the bike anyway since I (think I) can lighten it up same as I (think I) lighten myself up for curb/bunny-hopping etc...

With a touring or cross bike w/ longer chainstays, can put the pannier(s) well in front of the rear hub and the handling of the bike ranges from not affected to actually improved.

ps-> I was a committed rack/pannier person until I tried a roadbike. Now I am a committed fun bike person, and equip it however it takes to maintain the fun.