Yeah, movement of the outer race in a press fit assembly is bad as it can wear the press fit interface. While I generally let cartridge bearings run until they feel a little rough, I'd definitely just replace them in your case. 5k is decent mileage (though my BB30 bearings are doing way more than that, but they haven't been bathed in a river either). I would install using a bearing retaining compound in your case.