I don't think CF is absolutely less durable than metal, but it's pretty clear that one of the real problems is that incipient failure is not apparent. Everything looks fine until all of a sudden it doesn't. Also, the construction makes all the difference - perhaps more than the material. Bikes frames aren't made with cast aluminum, but if they were, you'd frequently see them failing catastrophically, too.
Which gets me to wondering: how does the failure rate of frames assembled from carbon tubing compare with the failure rate of molded frames?