Front brake.
Started riding fixed while living in Washington Heights - anyone who rides around there knows what the hills are like, especially going down to the Hudson River bike path from 181st. I've had a couple close calls where that front brake came through, including a toddler that ran right in front of me on the bike path near 72nd.
Now that I live in Queens and the commute is relatively flat, I definitely use the brake a lot less, but it's saved me enough times to warrant keeping it on. Maybe on a lower-geared bike (currently running 46/17).