Backpacks are a problem because they carry the load fairly high, making you more top-heavy, with more risk of falling over, and less maneuverability.
By that logic, it's a wonder anybody who's 6'3" and 300lbs can ride a bike at all! I'm a light guy at 130lbs and any backpack that I can carry while walking, I can carry while riding. Sure, 40 lbs of laundry on my back means that it takes more effort to start and stop. And more weight on the tires means one has to use a bit more care when turning, to keep traction. But in that case i'm still no worse off than I would be with the stuff in panniers.
For me, there are two significant problems with carrying stuff on my back:
my backpack doesn't protect the stuff in it from rain, and it makes my back get sweaty and stinky. (so then my backpack gets sweaty and stinky too.)