I find that drag doesn't really come into play unless there's a headwind or I'm going over 15mph.
I think we need more data before we can say the panniers are slower.
Is it just me, or is the nicer weather bringing about closer passes? I've been semi-buzzed every day this week after going most of the summer without any problems.