The feeling that the bike is more maneuverable without a rack & panniers is wrong.

If you carry the load on your back, the center of gravity (CoG) is higher, and moving the complete package of you and the bike is actually slower as a result, since the bike turns by leaning. Carrying the load as low as possible (i.e. panniers) is preferable for this reason. I don't like how the wind kicks my butt even more when I have panniers hanging off the side, so I try not to use them, but use a rack bag instead.

For shorter trips, the convenience of a backpack or messenger bag overrides the CoG and on-bike comfort issues.

Also, for the people who "prefer a backpack to a rack & panniers" I have to wonder how many of them actually really tried the rack & panniers instead of just judging by looks. At first, I thought the backpack would be the most efficient, but when I tried a rack and rack bag, I was hooked from the very first ride.