Goo Gone, Paint thinner, etc applied with some coarse steel wool and that residue is gone.

edit: the abrasive side of a sponge is even better.