I know what you mean. Most times, however, I'll take the opportunity to go, but I shake my head as I do so and mumble under my breath to show my annoyance. Whatever. It's better than drivers running the stop sign as you're crossing it.
I've also come to the realization through experience that because I appear to be someone who knows what he's doing, who knows the rules of the road, and who is decisive and assertive with his actions, that drivers generally respect that. I am predictable.
So when at the four-way stop I am given the right of way they expect me to go, so I go. Hesitation causes problems.