i almost never come into situations where something blocks me so completely and quickly that i have no way to avoid it. separate those into times when the quick deceleration offered by a brake would prevent a collision and times when collision is effectively unavoidable, and the probability of brake saving the day situations becomes somewhat remote.
roughly, when i'm hauling ass it takes me one revolution of backpressure to get to the point when i can skid/skip. i'd say that i can stop fast enough for most any situation because i never use my brake.
i'm safer riding with a brake, but i don't think i'm unsafe not having one.