This is an interesting topic, and I'm purely speculating here, but I think there's something to be said about people not really wanting to put in the work. Maybe some try and realize, like many of us do, that it's a lot of work just to be mediocre at this. With all the charity rides and centuries that different clubs host, I think that feeds enough of people's desires to accomplish something that they don't really try and do more. It's the same thing with running I think, where races have some competition among the more serious folks, but more often than not you've got people just happy to complete and not necessarily compete. In cycling, we've segregated the competing vs completing crowds (as it should be) but that may be some explanation for low turnouts.