It's the bike and the rider. Not the panniers vs backpack.
One thing that is slowing you down is that alfine hub. I know I'll get flamed for this but I installed an alfine for a very short period on my bike and it was noticeable slower by 2-3mph. Too much drag compared to a regular cassette. Maybe it would have eased up over time, but I really had to need for an internal hub gear setup in houston so I promptly went back to my pretty much maintenance free geared setup, and easier to change flats anyway.

But anyway, my point is there's lots of factors and the two biggest ones are the bike itself (setup, geometry, weight, etc) and the fitness level of the rider.

I wore my chrome messenger bag today instead of using panniers and I wasn't any faster. And I didn't sweat any more either but I'm in Houston and will be soaked from head to toe no matter what I do.