You did the right thing to keep yourself safe in the situation.
Who knows what that guy's problem is, but I bet he's got issues with everyone, including white folks. He just drew on his bank of racial hostility in your case. I'm sure he's got other sources of hostility. I agree with those who are saying to get plate #'s and a description if you see him again. You'll probably be doing someone else a favor, as well.
BTW, I'm "white" (if there is such a thing) and lived on a reservation at one point. The Native Americans used to tell
me to go back where I came from. Kind of puts his stupid comment into perspective.