I've seen enough plastic and rubber get hard and brittle with age to know with certainty that old rubber or plastic can break unexpectedly when stressed. Like when you're trying to remove an old friction-fit part by pulling or prying on it.

Your bike, your potential future problem. Do what you want.