At age 20 you have all your faculties and can actually do the things the racers do. They can do interval training, etc. But at age 50 + its a far different thing. Just imagine how much time it would take to race competitively, how much work it would take to do the interval workouts, the hill climbs, the individual time trials. There's a lot of preparation to get an athlete to racing condition.
Not only do you have to carve out time to do all this but you also have to figure out if your body can take the punishment, the mileage and tough workouts ahead. Those of you who ever were in competitive sports know what I'm talking about.
Otherwise we would be like those silly bowflex commercials, all claiming to be grandparents and buff.