Elitism?
Just because someone, through interest, hard work, and love, is able to do something difficult and rewarding? Be it riding fixed gear, telemark skiing, sailing, or using Linux. Being an expert with math, science, writing. Being an artist.
Folks who use the term 'elitist' negatively are probably envious.
Positively used, "Fixed gear riders are the elite of bicycling."