Rubbing on any surface will cause friction and wear the surface down, even leaving a water hose running across pavement will eventually wear the pavement down. So adjust your cleat, (if you haven't already) away from your crank and ride on.