Sheldon Brown has written extensively and collected the thoughts of others on this topic:
http://sheldonbrown.com/fixed/index.html
The allures of the single speed for me are the mechanical and psychological simplicity and the perfect chainline. Once you feel that chainline, it is difficult to go back. I initially converted an old mountain bike to single speed and then I bought a fixed gear road bike. I always feel that I have three gears: in the saddle, standing and walking.
Finally, when climbing I must admit I do get a kick out of dropping the hammer on a color-coordinated peleton of multi-geared riders. Alas on the descents, it is dangerous to compete.
Forget all that, it is "a cult thing."