Try some rubbing alcohol or fingernail polish remover. Works well on sticker residue as well.
Nothing wrong with wanting to keep your wheels clean, but eventually the paint is going to wear off and there's really nothing you can do about that.