That was one of my first thoughts, but I didn't have a magnet strong enough. And by now, I've got the piece wedged in there tight enough that I don't think it would work.

Another thought I had was somehow gluing something to the broken ball and just pulling it out. Is there any type of glue that sets quickly and bonds well to steel?