I definitely understand where you're coming from but you can't deny how common it is for other types of cyclists to treat fixed riders like we're doing something crazy, stupid, primitive, wreckless, etc.
I think that in itself makes us riding fixed feel like we're on the fringes of something - whether that's true or not.