No, that's not so.

By far the biggest health gains come from making the change from sedentary to moderately active. You don't have to be really "fit" to be healthy. And it is true that after that there are diminishing returns. Doing twice as much, and being twice as fit, doesn't make you anything like twice as healthy. But it does tend to make you healthier. At the top end of endurance athletes there seems to be some evidence of increased incidence of heart arrhythmias, and that has got a lot of publicity. But we're talking about very high volumes at fairly high intensities, and even then, it doesn't seem to dent the fact that people who exercise intensively tend to live longer than those who don't.