I just spent a week or so riding around Ohio. For a short leg I was carrying more junk then could fit in my panniers, so I supplemented with a backpack. Not bad for 20 miles. I kept the backpack with me the rest of the time and kept all the stuff in it that I didn't want to leave on the parked bike, but unless it was a short jaunt, I always shoved that backpack into a pannier. It probably only had three pounds or so in it, but still I'd rather ride without it than with it.

I agree that a light enough pack doesn't make a huge difference, but I would still prefer the option of not wearing it.