Ever hear the phrase "never bring a knife to a gunfight"?

This applies to pepper spray also. The folks that represent a real danger to you won't threaten, they'll simply brace you when they have the opportunity. And, they'll most likely be armed with a knife or gun. So, unless you're super cool, and super fast and accurate in a crunch, odds are the pepper spray won't work out in your favor.

OTOH - those nuts that are all talk may cause you to take action that isn't called for, (in the eyes of the law) and depending on the jurisdiction may cause you legal trouble.

So carry the spray if it makes you feel better, but IME the best defense for the sort of thing you experienced is to walk purposefully and project strength as you move away.