It never happened to me, but it's NYC, lots of people ride bikes here. I also dress up in cycling clothes and my bike is equipped with all kinds of gear that probably makes people thing I ride on purpose.
Only once a guy in a car yelled at me "get a car!", but that was a long time ago and I explained to him that I probably make several times as much money as he does, I'm 1/4 of his size, a lot healthier, smarter and generally don't give a sh*t about what what dumbass fatty thinks and rode away.