The issue that usually comes up is that there's just not that much material that makes up those tiny cable guides, so if you are not careful, with taking off the rust with mechanical methods, what's left of it under all the rust might not even be enough to support the brake cable without eventually breaking up, especially the most forward one......
So if your bike starts rusting at the TT cable guides, it is critical you give it some attention asap, before it gets too bad to fix....