As fietsbob brought up, clean air vs. already in turbulence, or directly behind (part of) the wind punching object's profile, makes a big difference how much drag something will create.

Panniers are right behind a pair of thrashing legs, I doubt there's any clean air for them to push against.