If it were me, I'd leave the rest of the paint. Might help to protect the steerer tube from rust…a bit.

I concur with the carefully used paint stripper idea. Better yet would be someone with the crown milling cutter, and who knows how to use it. If the crown race that was on there was so tight, maybe a run through the mill might help some.