Let it go and don't let it ruin your day.
I listen to an ipod at it's highest level and usually don't notice people harassing me. I've been passed close enough to be hit with mirrors but once I'm passed there's not much I can do.
A guy in a truck threw a bottle at me once but he was doing at least 40 and missed me by 50 yards. Someone else threw a cup of soda on me once. He was going slower and was able to get close enough to spray me.
When I first made the transition from a car to a bike, I was unaccustomed to having people f*&k with me and take advantage of my vulnerable (and much slower) position, so when someone did something dumb that endangered me, or when they insulted me or threw stuff at me just out of the blue for no reason, it really pissed me off because these are things people never did to me unprovoked for the whole 15 years I was driving a car, so I wasn't used to it.
But then you realize there is a never ending stream of idiots, and cars stuffed full of teenagers, and that will never change. If I am not harmed, and my bike is not damaged, then it's not worth it ruining my day. Hearing about the 3rd cyclist being ran over and killed in my city this year (and not a big city) helps put things in perspective, and reminds me not to sweat the small stuff. You have to stay humble to stay safe. A big ego will put you on the offensive, and there is no reason to be offensive when it's a 20 pound bike versus a 2 ton car.