Yes, definitely
I have noticed a gain in speed when I use my hydration-pack only, as opposed to both hydration-pack and pannier. Then again I tend to just pack the pannier to the gills full of things I don't really need or use on a daily basis but I feel comfortable just knowing it's there.
Ultimately, it's not worth the extra weight, drag, and lack of a fun bike ride. That's why lately I've been trying to pack lighter, more minimalist, and use only the pack which has 1600 cubic inches of space.