I don't ride with anything less than 70 GI in San Francisco. Sure, it makes it easy going up, but going down is a different story.
With 73 gear inches, I'm already spinning downhills. Some of these hills will make you spin out with very much less without a brake.
2. I was in sf the other day and spotted a few FG riders. How in the world do these guys have the balls to be flying down these hills with no brakes into busy intersections. I've gone down some steep hills, but I'm pedal pacing if I need to stop soon.
I can't speak for those guys, but I can say that I try to ride with a 'flow.' You need the momentum for the hills. It's not recklessness if you're not stupid. If I'm flying down a hill and I see a controlled intersection upcoming, I pedal pace slightly to time it so I get a green light. Use your best judgment. A lot of members here describe riding fixed as becoming more aware of our surroundings and 'looking into the future.'
Final note, keep that brake on. Sounds like you need it, and there's no cred lost if you have it.