I think your first instinct might be right - don't load it! Or, how about a camel back or small backpack? I've seen a lot of riders on supported tours with the seatpost rack & trunk combo - I've never used that, though, so no comment. I guess it really depends on the weather - how much extra clothes will you need during the day, over and above the options you stuff in your pockets on your local rides. Oh yeah, and how big your camera is.
Rowan... I really don't see where your speed/weight-weenie comment came from in the original post... or maybe my humor-detector isn't working?
Peace,
Anna