Ghrumpy

It is the case that caged bearings allow for fewer balls safely to be used in a given assembly, and that it can speed assembly.

Admitting that, however, I seriously doubt that Campagnolo, for one example, had either cost or speed of assembly in mind when they decided to use caged bearings in their Record bottom brackets and headsets way back in the 1950s. (It is true that their hubs have always used loose balls, as do most hubs.) And Campagnolo BBs and headsets use a full complement of bearings. Removing the retainers will not increase the number of balls you can fit in them.

So I'd submit that it doesn't have to be one reason OR another; there can be multiple reasons for retainers. The friction reason for them is, in fact, to keep them from touching each other. That's because sliding friction between lubricated steel parts is literally 100x more than rolling friction. (see http://www.tribology-abc.com/abc/cof.htm. The coefficient of sliding friction for lubricated steel is 0.15. The coefficient of rolling friction for steel balls is 0.0015) Friction reduction is the entire reason for ball bearings in the first place. Now, since the balls themselves are rolling in the same direction, the points where they touch will not just be sliding against each other, but also at twice the speed they are rolling. If the sliding friction becomes high enough, which wouldn't take much, it will stop the balls from rolling in the races momentarily, which will quickly damage the bearings and the races.

FWIW, the mangled retainers I've seen appear mostly to have been damaged by the failure of one or more balls, not the other way around. Balls are much harder than retainers, which are quite soft as steel goes. Or, just as often, they were installed the wrong way around in the assembly.