If I were you I'd try a little investigative poking around, like with a piece of wire or an old spoke or something. See if you can detect an inner guide tube or not inside the frame tube. Some bikes don't have it, but that should be readily apparent. It still sounds to me like there is a sharp burr causing the damage you're describing. I know that there are different housing diameters for shift cables but as far as I know brake housing is all the same. Don't use a shift housing for a brake cable, BTW.
If there were meant to be cable housing stops on the tube, that should be easy to see. Unless the previous owner did a little "modifying" with a drill.