Pedestrian right-of-way in the US is a state or local issue. Some places the ped has the right of way 100% of the time. Other places it's ROW when in any crosswalk. Still others it's when in the crosswalk w/o seperate ped signals or when in a crosswalk with signal showing a walk.
Here it's basically any time the ped's in the crosswalk (basically...it's actually more ambiguous than that).
Still, if I have the ROW and someone decides to step off the curb, I'm not going to hit them, but I'll play a little game of chicken. Far too often I have exactly what you describe happen. They know they would get creamed by the car, but they figure they got my number.
Well guess again pally, I'm doing 25 mph and I'm not moving. You really want to see if I'll stop? Maybe you should just wait your turn.
Oh, and be sure to give them a holler like, "look out!" so they know you know what they're up to.