I'm in the same boat as you, bringing work clothes, lunch, lock, and police-style boots in two saddle-bags. I get passed a lot as well.

I'm guessing you would be faster with the backpack's lower wind resistance, but the best thing to do would be to test a backpack out, and see if it is worth the back sweat. Just a heads up that the packs I've seen for cyclists are fairly narrow and may not fit your laptop. But, that's where bike messenger bags come in.