Hmm, PVC pipe sounds like a good idea, but only if you can find a size that bears down directly on the race...
Otherwise, if washers are necessary, I'd make em out of plastic.
If you can't get it all the way on, you could ride it for a bit to hammer it on the rest of the way - as long as it's already 2-3mm of the way on, and dead straight.