If you put holes in your frame when you are trying to remove cable guides you are doing it wrong and I take them off all the time... of course, I only remove them because they are damaged and replace them when we are repairing frames or because some doofus filed al the guides off a really nice frame and then retired it.

If is is a really nice road frame it would be a bad idea to remove them as one day you might decide that you want a geared bike and then those handy cable guides will be gone.