You did fine.

Another tactic is to pull your cell phone out and start calling (or pretend to call) 911. Maybe hold the phone up like you are letting the dispatcher listen to what the guy is saying.

It is amazing how fast these bullies start running away when they think you have the police on the phone. I am convinced most of them have been arrested before by their reaction.