I don't know anyone around here that rides fixed gear or brakeless, but I can't imagine anyone giving someone hard time for having a brake.

The guys I do the Saturday ride with think it's kind of crazy, but they don't make a big deal about it - even the dudes that are super precautious and yell "CAR LEFT!!!!!!!!!!!" when they're by themselves.