About the only time I remove old glue residue is when the previous owner has taped up to stem, as I like to have about 5-7cm exposed on either side. Sometimes, I'll clean further, if the the correct OEM configuration is half tape or grips.

Often, if the previous tape is cloth, I'll just tape over it, as this adds some additional cushioning and builds up the diameter slightly,adding to comfort.

I hate the way cork tape rips and leaves small bits of actual cork, as opposed to just glue residue. These I do clean off. It's a Princess and the Pea situation for me.