My wife used a pair of Ortlieb Packer Plus small front panniers on her rear rack and her sleeping bag and Therm-a-rest pad in a dry bag on top the rack for a 74 day ride across the U.S. She probably had 30 lbs of gear. Both of our loads combined weighed less than 70 lbs. I'm sure you could pare your load down, and still be comfortable.

P.S. She also carried a bar bag.