On a charity I passed a guy that was standing on the side of the road scraped up, and his helmet in pieces. Perhaps if he didn't have a helmet he wouldn't have been standing. I've endo'ed twice on one ride, my lid had had a few scrapes and cuts, better that than my head.
Do what you want, I'll keep wearing mine.