If you need to take the stuff with you, does it matter?
When I switched from a backpack to panniers, I don't know if I was slower or faster...but I FELT faster, simply because I wasn't holding so much weight on my back. If the aerodynamic drag slowed me down in actuality, I don't really care.
__________________
Good night...and good luck