No, but if you have a problem riding with a backpack it probably means you have the wrong pack. Get a narrow fitted pack that's comfy. I do my 14 mile commute with a pack. Yeah, it's a bit hotter, but I sweat whether I wear the pack or not.

And I pass guys with panniers all the time ; )