Whatever works for you is golden.

In my experience, the amount of stuff I carry tends to be related to how much convenient cargo capacity I have. In other words, rack, box, panniers whatever tend to accumulate items which I sometimes want and like to have. While conversely having less capacity I tend to work out alternatives.

Accordingly, to pack lighter just start reducing capacity and adapt. Take the panniers off, and work out how to do without something or do it differently. Then put a smaller bag on the rack. And so on. I haven't reached the extremes of pulling a trailer or racing to work with nothing on a daily basis, but I have varied between carrying a lot and very little, finally settling on a small pack with the daily change of clothes with a few toiletries.