I never keep the bearing retainers when I do rebuilds. When you take the loose balls out and install them, you'll notice that you have ample space left for additional bearings, which means more rollers on the races, which means less wear on the surfaces. Plus, so I don't have to remember which way the race faces, I make sure I only use uni-directional bearings