Personally I wouldn't be too worried about the frame. Even if it does eventually fail it seems likely that it would do so with some warning. I agree with the idea of sanding down the area for a closer look. I'd also take the fork off and give it a very careful examination. I've had one experience of a sudden steerer tube failure and don't want a repetition.