Yep that was a ball retainer.

You could dispense with the left retainer but run the risk of the balls dropping into the hub. I think the left is a standard size. I ran mine for a while without left retainer.