I'm with the OP - most of the folks who honk, act up and otherwise annoy me are lower on the socio-economic spectrum and typically seem to be lower on the education/intellect sprectrum. They're also usually, but not always, young. I don't get hassled in Center City or nicer areas, I get hassled while driving through crappier areas. The last time I had an issue with a nicer car, the driver was a thug, just a thug in a nice car. I should say that I rarely ride in the burbs, so my experiences are probably skewed. I think Center City folks are used to bikes here and I think we have better infrastructure keeping each other out of one another's way more often. I think lower in town speeds might play a role too.
My guess? You have pissed off, resentful people with someone they can bully.