Okay, now I think I understand better. I wouldn't want to sand or pick away at anything embedded in the paint on the downtube either. Perhaps you could take a piece of an old thin inner tube and cut a gasket that will make the annoying gaps disappear.
If you do sand, then probably should at least clear coat it in that area. If you don't sand through the paint entirely, the clear coat will or should bring back much of the color and it'll not be too noticeable.