Right, that would be insufficient reason. But that's not the significance of this concern over rotational injuries in my opinion.

This concern is based almost entirely on the concussion portion of traumatic brain injury. You can stipulate that helmets can save you of most of the abrasion injuries, and contusions and other effects of less severe impacts, and I doubt that you'd get much argument over that. It's reasonable to decide to wear a helmet on the strength of that alone, and also reasonable to decide that preventing those injuries alone is insufficient reason to justify wearing one.

But concussions are serious concerns, and it's pretty startling to learn but solid that the most likely cause of concussion is rotational force, not linear impact. It's also pretty startling that helmets provide little or no defense against those rotational forces (although more recent helmets improve on this), and possibly, no protection against a large range of the larger linear forces that are required in order to cause a concussion.

So there IS a range, maybe, where the maximum impact that the helmet can absorb and the minimum possible impact that can cause a concussion overlaps and hence the helmet might save a concussion. Even that is disputed, and if true is a narrow range. So that line of thought basically has it that cycling helmets offer no protection, or at best little protection, against concussions. But they might increase the chances of a rotational injury.

From that reasoning above, if a person accepts the (small) possibility of the relatively minor scrapes, cuts and bumps, then the helmet may be worthless or less than worthless for the relatively serious concussion injury. I think that's really the main reason people say that.