I used to find more wear on bike shorts on the starboard side, just below the crotch. Then I noticed that one leg tended to flop inward almost brushing the top tube. I've made an effort to maintain knees the same distance from the top tube, with a bit of help from a cleat on the shoe on the same side. Over time the asymmetrical wear pattern has disappeared.

I would say you have some sort of asymmetry that needs to be dealt with. Perhaps one keg is very much stronger than the other. Good luck with this.