I disagree.
Those bikes are built to take much more than you can ever dish out.
Seriously, a decent set of wheels and tires will get you across any surface.
In attempting to avoid dangerous traffic, I've ridden my Trek down washboard roads, cobble streets, rail-trails, etc. at full tilt.
Granted, you might have to replace headsets and bottom brackets a little more frequently, but you're not going to kill your bike. Ain't no way.