I do race, but I think it's perfectly reasonable for most people to not want to race. Mass start bike racing requires a very different skill set than riding in group recreational rides, and carries a lot higher risk than those rides. It also sucks up a huge amount of time-- you really can't just race once in a while, you have to race pretty regularly and focus a lot of your riding on being prepared for it. In running races most people are trying to get a better time than last time. In bike races (mass starts) the time is irrelevant-- just the order you cross the finish line matters.