Because the UCI outlawed disc brakes. That's the real answer that no one ever wants to say.

Without those, since dual pivot calipers don't offer enough clearance...and v-brake/travel-agent setups can get kludgy...you work with what you have left available to you: Cantilevers.

While UCI rulings don't actually affect the larger cycling population, they do in that every-day joe riders tend to follow them; so the brief explosion in disc equipped cross bikes died with that ruling. But for a few years it looked like the "next big thing".