Until you start approaching the UCI weight limit (~15 lbs), road bikes are pretty durable. Wheels are probably the biggest weakness, but until you get really spendy, most road bikes come from the factory with a pretty durable (albeit heavy) wheelset. Avoid featherweight, low spoke-count wheels and you should have no problem. Check out some cyclocross videos for a more realistic idea of the kind of abuse a road bike can endure. CX bike frames are typically about 25% heavier/stronger than pure road bikes, but they're designed to spend close to 100% of their time being thoroughly abused.

When the going gets a little rough, wider tires will dramatically improve the comfort factor. Finding a road bike that can accommodate a 28 or even wider tire might be a wise move. Some of the race-oriented frames struggle to handle more than a 25 width. Cyclocross bikes can typically handle up to a 35 wide tire, and gravel-grinder/touring bikes can often go as wide as 45. My "go to" bike for all but the most intense training rides is my CX bike with a set of 28 slicks. Comfy, bullet-proof, and only a couple MPH slower than my carbon "race" bike.