Generally speaking you should replace them if they make noise, or feel rough. I will generally let them run if there's just barely perceptible amount of bearing feel. Generally the replacement point should be before you notice the friction at the cranks--there's obviously a lot of leverage at that point. Spinning the outer race is a pretty clear sign of both bearing wear and the need for a better interface between the bearing and the shell.