So when you see me leave 3 seconds before the light turns green, after I've double checked there's no more cars turning left in front of me or behind me or to either side....don't be in a hurry to catch up just to tell me that I am "supposed to wait until the light turns green"... I really don't care what you think...
If you really don't care what the other person thinks, then why are you getting bothered by his comment?
You don't have control over the actions of others - including comments they might make about your behavior. But you do have control over your own reaction to those comments.