Just for his attitude, I'd file a lawsuit on the guy. You might not win, as you said, your word against his. But at least you have the satisfaction for dragging his sorry behind down t the court room. Should only cost you about $10 and your time.
He is really setting a bad example for his son.