What do you mean by chewed up a small amount? Is the race damaged where the bearings seat? If so, yes, of course that could be the problem.

The other problem could just be that you are not adding enough preload to the bearings. I'm telling you, they need A LOT.