If you want to win, you might need gears, but you can totally race and have fun, and probably even do pretty well, on a singlespeed. I found myself using one gear most of the time last year on my 3x9, so I'm thinking about racing singlespeed this year. Of course, around here we have a special singlespeed category.
Also, even this question is answered on the Sheldon Brown site (though not by Sheldon):
http://sheldonbrown.com/singlecross.html