Next time I'll use tape! Thanks for the suggestion Looigi.
A bigger magnet might pull it out of the spindle well. If that doesn't work I'll try to epoxy something to the magnet. If that doesn't work, I'll try to drill a hole and then use a small hook to pull it out. Until it's a problem, it's not a problem.
I mostly wanted to warn others, and to say how well the rare earth magnets work. I even added one to the spoke magnet that triggers the speed sensor. With the rare earth magnets, the adjustment tolerances are very wide and easy to set. I don't know why these sensors don't come with rare earth magnets standard.
For those that want to use a rare earth magnet on their spindle, buy the kind that has a hole in the center so you can pull it back out if necessary.