Both my road bikes are Ti. So, yes, I'd feel comfortable taking either for shorter distances on gravel. But I have a mountain bike that goes on the gravel roads. I'm more concerned about how narrow tires would handle on gravel. And the dust and grit.
If I had a CF frame? Nope, not unless it was provided for me.