Yeah, that makes sense. Textured surface isn't going to sand well. From my limited understanding of rust converters, that brown is probably iron tannate, which is OK to paint over. Or it could be some fresh iron oxide. I'd still try to remove it.

I'm thinking naval jelly and a wire brush would work. Maybe some scotchbrite. Test first.