I've found that even if I do get the plastic cap off a few times, at some point it is going to collapse.
And every time you have the pain of remembering how you got it off last time.

Nowadays I avoid the pain by chucking the plastic cap and getting some metal ones with hex holes, I've never broken one of those.
Plus you can polish them.

You can even get Sugino ones:

Original plastic:


Replacement metal: