Instead of touching each other at one small point like with loose bearings, what do you think the balls in a cage are rubbing up against? The contact area with a cage is much, much larger... more friction... Also with caged bearings, you have fewer bearings, so each one will have higher pressure at their contact point for the same load (compared to a higher number of loose bearings). This will result in increased wear.

If you hear clicking in the bearings, maybe it's time to grease them... The blue boat-trailer bearing grease works really well to quiet any bearing noises.