Wait till it wears out or starts grinding. It won't harm the frame because it's set into headset cups that are typically removed and replaced with a new headset.

DoctorBike