I think the perceived threat is what prevents people from racing as much as the real threat.
But there's also a fear of being humiliated: dropped. Left behind. Like I've said elsewhere, you can't get so blatantly dropped out of the game in other sports like you can in road racing.

The sport is hard. There's a selection before the race even begins. Heck, even before registration.