Don't be afraid of other people's impatience at a stop signs or red lights. Don't let their impatience cause you to disregard your own safety. The best thing you can do is don't engage these a**h***s either verbally or with gestures. Move to the far right and let them pass you before you clear the intersection if they're being rude. I've had several experiences like yours at stop signs/lights here in Portland, mostly in the 'ghetto' of Portland (anything east of 82nd Ave). The only way to deal with it is be calm, courteous, and do not engage them in the way they'd like you to.

Yes, this is one of many reasons I roll through stop signs and red lights when it is safe to do so. However, as you mentioned that this is:

you clearly made the correct choice coming to a full stop and waiting for the traffic to clear. This is what I would have done as well.