This happens all the time to me too, but I NEVER take drivers up on it unless I'm trying to make a left turn and there's a huge line of traffic backed up in the oncoming lane. For me, it's a safety and liability issue. I prefer not to insert my 25-pound, 1/2-horsepower vehicle in front of their 4000-pound, 350-horsepower vehicle when it doesn't belong there. I also don't want to be held responsible for damages from a crash because I violated someone else's right of way. Yes, they often get pissed off and swear and yell and burn rubber when they take off; I'm not sure what sort of issues they have going on.
If I see a driver stopped at a 4-way stop that I'm approaching and they seem to be waiting for me to run the stop sign I'll slow to a crawl and roll up to the line veeeery slooooowly. If they still haven't moved, I'll stop, dismount, and pretend to inspect some component on my bike until they go.
My favorite dumbass motorist situation is this one: I'm signalling a left turn -- a driver in the oncoming lane stops to try to get me to turn in front of them -- I don't turn and after a while they finally move on -- the VERY NEXT driver stops and does the same thing!