I agree about the handlebar bag. It's really nice to have a few things handy, and not have to dig into a pannier.

Panniers don't weigh that much, do they? I have some big panniers for the rear and smaller ones for the front. I'd assemble all the stuff you're planning on bringing and see what you need to carry it. If you end up with large panniers that aren't all the way full, what's the big deal? You'll have more room for souvenir t-shirts, fine cheeses, maybe some bags of chocolate bars......