Hard to say for sure, but I think I'm faster with a pack on trips of 7 miles or less. Probably could get a better time with panniers once the trip gets up to about 18 miles. 7-18 mile trips are about the same whether backpacking or saddlebagging.

Backpacks make stomping on the pedals more fun, bunnyhopping potholes and curbs 10,000 easier so that's where the speed comes in. I don't think aero means a thing unless you're really haulin' butt or in a harsh headwind.

OP, I think you're getting passed cuz you're slow. 'salright, I am too.

Today I had to ride about 8 miles with a gallon of orange solvent, a quart of glue, an 18"x100yd roll of tape and 100 big sheets of transparencies. It was really too heavy for the backpack. My butt was starting to split in two by the time I got to the shop.

So, you can see that load plays a part. Ain't no way I was gonna be bunnyhopping anything with that on my back. I still sprinted off the line at a couple of lights, tho.