I'm going to wager a guess that you have taken your disc off, DrPete.
Whether they are truly elite or not isn't the point. Some people view themselves as being elite because they don't use the dork disc, and consider themselves the better cyclist over someone who still uses theirs.
"You have a dork disc on your bike? I must be the better cyclist." That, my friends, is the point. It's simply an untrue statement.
With this attitude a bit more common then it should be, the laws of chance dictate that sooner or later you are going to pass up one of these "elite" riders. I've done my fair share of passing up other riders (and I'll freely admit being passed by others). The dork disc did not figure into our abilities at all.
It's the engine that makes the bike go. No amount of dork disc removing is going to significantly improve your race times. What's the advantage? Taking off a few grams of weight?