Dry 3M Scotchbrite scouring pads on the rim. It takes off everything w/o harming the aluminum.
(Easiest if the tire is off the rim.)
Assuming the pads are still usable, just rub them across fine sandpaper to remove any glaze.