As much as possible, I aim to keep the weight of the stuff I'm carrying down to half my body weight. I've mentioned this very rough "fomula" here before, and it seems to work for me. When the weight of the bicycle, my panniers, and everything I'm carrying goes above half my body weight, I struggle. I can still ride, but it's quite an effort.
One of the things I do to keep the weight of my stuff down is to use only 2 panniers. I see people with 4 panniers, and I'm not sure what I would put in 2 additional panniers.
I also know that in many instances I can buy what I need along the way. I don't have to start the ride with a full set of clothing for every occasion, I can buy a sweatshirt or wicking T-shirt or whatever as I need them.