Yeah. I've ridden my CF road bike on dirt, gravel and even many miles of single track. The limit is the tires and possibility of pinch flatting more than anything else. Next is damaging the wheels by hitting a sharp edge. The last thing I'd be worried about is the frame.

Cue a Marty Ashton's Road Bike Party vid....