Have to agree. I've been in your situation before, and I gave the guy my best salute. He immediately slammed on his brakes, and tried to grab me off my bike. Scared the S*** out of me.
As much as you want to tell someone off, the next guy could have 100lb. on you and thinks he's the UFC champ of the world or something. Worse, he could have a weapon and no conscience about wanting to use it.
I know that it's easy to say "stand up for yourself" but the reality is just ignore it. Ride away, swearing under your breath if you need to, but it isn't worth it.