I go on a lot of midnight rides where we have a pretty mixed crowd and on several occasions I have had new guys tell me there was no way I would be able to keep up running just one speed.
My response has been to simply tell them to stay out of my way on the climbs and I have even dropped some of those gearheads on some pretty insane descents... I rock a pretty high gear sometimes.
All in all... it is nearly impossible to explain the attraction to riding a fixed gear to anyone who hasn't done it.