You don't mention how far you are riding, or whether your bike is set up for panniers (does it have a rack?)

I would definitely say this is not a 'one size fits all' subject.

When I started, I used a backpack because it was cheap (I already had one), it was convenient (I am multi-modal, and carrying a backpack off the bike is tons easier than carrying panniers off the bike) and it was simple (no extra hardware, nothing to mess with). The backpack was great when I was riding a mile and a half each way. It was fine when I was riding 6 miles each way. It was not so cool when I started riding 10 plus miles each way. Note that I was not using a cycling specific backpack, and it probably weighed in somewhere between 12-20 lbs most days. A well fitted backpack designed with cycling in mind should be a very efficient way to carry stuff, and I doubt one would experience 'sway' on any decent pack. But, if it isn't well fitted, the straps are likely to impede mobility, and there could be issues with excessive motion. My pack is not cycling specific, and the straps are a bit too wide for riding comfort. I don't notice on short rides, but after 40-50 minutes, I notice the tension in my neck and shoulders.

Having said all that, I now ride with panniers. They are definitely comfortable for longer rides, insofar as the weight is not on me, but I do feel they exact a much greater toll on the bike's handling and feeling of stability, esp. when I began using them. Like most things, you adjust, but this might be an issue depending on your riding style. And, although the weight is carried lower, it is also carried to the rear of the bike (unless you plan to do front and rear racks and bags), which makes my bike feel light in the front (not always a bad thing, of course). And, if you have to pick up or carry your bike, panniers are a bit of a pain, as your 20 lb bike now weighs 5-X lbs more, with the addition of the rack, bags and load. And generally, that rack is there whether you use the panniers or not, which is added weight to pedal around all the time. There's also aerodynamics to consider, but that's the least of my worries :-) It also takes longer to get the panniers off the bike than it does to walk away with my backpack on.

The sweatiness issue is not a major factor for me -- I sweat plenty either way, and all that changes is where the sweat accumulates. Your tolerance might be different.

I definitely have more space with my panniers than most cycling friendly back packs offer, but that isn't necessarily a good thing, as I tend to carry more junk because I can. I was lighter and leaner without :-/

My take is longer rides? Panniers are far superior. Shorter rides? backpacks might have real advantages, and even if they don't, panniers might not be worth it.

I've never used a messenger bag, but no matter how well designed, I have trouble seeing how it could possibly be more stable than a backpack. I think that with a waist strap it should be as stable as a backpack with only the shoulder straps. It does move the weight lower, which might have some advantages. I tried riding once or twice with a regular shoulder strap briefcase when my backpack zipper blew out, and found that nigh unto impossible, as it wasn't stable at all. I do NOT believe any well designed messenger bag would share that flaw.
Last: As with most things, you probably get what you pay for most of the time. Cheaping out (as I usually do) is probably a false economy, if you plan to use the stuff for the long haul. I have middling cheap panniers because I wasn't sure I'd like them. When I need to replace them, I intend to move up to high quality bags. The same is true of my pack, however: it was a low cost alternative when I had no real money to spend. I would guess I'd be more pleased with a better fitting, cycling oriented pack.