In addition to chewing up the stays, they add weight. For utility bikes this is not a big concern but if one goes through the grief and expense to get their racer under 20 lbs, a kickstand is an unwelcome addition.

I will concede though that on bikes that have a dedicated kickstand mounting at the chainstay bridge, they are not such a bad idea and offer convenience w/o damaging anything. I kept one on my '74 Moto GJ and '83 Pug UE8 for reason. (Neither are especially light weight bikes)

An alternative to kickstands was Rhode Gear's "Flick Stand". These simple little devices held the front wheel stationary so the bike could lean safely, were lightweight and never scratched.
I think the evolution of downtubes of varying size and shape may have stunted their chances for widespread acceptance though.