More than the shirt, there's the wet tent/tarp thing, which will get your sleeping bag wet pronto. I keep my sleeping bag, tent, clean clothes, dirty clothes, and warmer clothes, all in separate extra small dry bags, which then all go into a large container, in my case waterproof panniers and a rack-top bag. You already have a rack. Pacific Outdoor panniers only add 2 lbs. The extra long-term comfort might be worth it to you. The wife and I tandem toured with 40 lbs., including pump, tools, rack, bags, etc., etc.

I camped with a guy who rode to California from Florida with a camp chair in his gear. He did 93 miles the day before I spoke with him. The chair was worth it to him.