It's been a couple of months since I last commuted with my messenger bag. My new and totally inappropriate for commuting tires came in the mail yesterday and I wanted to give them a good run. So for today I left the panniers at home. I even took off the rack. I didn't go quite so far as to don a jersey for my morning commute but I was tempted.
Yes it makes a difference. Accelerating from stops, sprinting or just putting your head down and going. Something a little interesting: I usually stand heading away from intersections. Having the extra weight on you instead of the bike at this point gives you some extra power, but I also seemed to get tired quicker while standing than I normally would. I had a hard ride late yesterday too so my legs were already a little tired and that might have had something to do with it.
Now, a lot of people don't ride like that while they commute. I don't want to get into a debate about how right or wrong any given riding style is. But whether you prefer backpacks or panniers I would guess is largely dictated by your riding style.