Do you tuck your pants into your socks, roll them up above your calves or use pant clips or straps? If not and they are loose fitting, they may be occasionally touching your chain through your frame.

Do one of the above if you are not already. Catching a pant leg on anything while you ride will at the very least age those pants a lot. It could well cause problems just riding the bike.

Ben