IPA isopropyl alcohol or similar should remove the residue without affecting the paintwork, stronger solvents like Acetone could damage the paint and any rubber it comes in contact with, although a dilute solution of this as found in nail varnish remover should be safe.