Taking off the remaining glue on the surface is not that difficult. Use another piece of duct tape, press it and lift then repeat several time. It's usually comes off easily depending on the texture of the surface and the shape of the surface and age of the glue. Just make sure your fingers are protected in some way, or otherwise your skin comes off and it hurts so bad.
I used two layers of duct tape behind the tire wall as a boot to protect the tube from sticking out the hole on the side wall. And when I mentined it in here, someone told me that the glue makes a hole in the tube(from degradation?). That's a big possibility but I have been too lazy to replace it. As someone suggested, electrical tape may not be a bad idea.