The worst the structural damage might likely do is just let it open up and flat again. But every mile you get on that tire is a mile less wear and risk of puncture for any other new tire you put on it. If it makes a bump every time it goes around and just annoys you, then that might be a good reason to change it now.

You should see the big pooched up place that plugs they put in car or truck tires sometimes makes.