The reason why we get pissed at drivers is basically the same reason why driver gets pissed at some of us. We are not consistent, and did something unexpectedly.
Though I suspect that people have loud complains about cycling not because there's more of them, but as a group, we are less consistent. e.g. we can expect most drivers to stop at lights, we can expect that most drivers will stop at at stop sign, from a small road to a major road, and be mindful of californian stops if it's a 4-way stop in the middle of the suburbs.

But different cyclists do different things. Some stay on the sidewalk, some stick by the rules strictly, some bend it a bit (guilty as charge, at stop signs with clear visibility, I will slow and roll through if I know there's plenty of space), and some are just outright reckless.

Tackling that problem would be interesting, yet tough.