It's not about breaking the bike, it is about breaking the rider. Seriously, the bike can hold up with exception perhaps of very lightweight wheels and tires. The problem is the rider's comfort and ease of pedaling, balancing, and bike handling on surfaces rougher than normal roads. Not everyone has trouble with it, but riding a full bore road bike on gravel, sand, mud, or grass can be challenging for some.

See the similar thread established over the last few days. The posts there may help you.