Get some 80 or 120 grit sandpaper. Get a rubber eraser or other sort of flat - but flexible - stuff. Wrap the sandpaper around the block. Sand off any high spots. Grooves in the rim won't make much difference, so don't worry about them.

This should not weaken your rims significantly so don't worry about it.