I'm the same way, but it's still possible to cover all bases for 3 season camping with well under 20lbs of gear. And if your load is that light you can also ride a bike that's in the 20lb range.
+2 on this. I put my three-season stuff for a two month trip in a four pound set of panniers and weighed in well under 20, more like 17. I wish I had a lighter bike--the heavy touring bike was overkill, but it was comfortable and durable. Same thing with the panniers--too heavy for the load but It's hard to imagine lighter packs lasting as long and being as easy to pack.