I've gotten road bikes with 700x23 tires down more miles of dirt roads -- plus three very memorable miles of dirt road in pouring rain -- then I care to remember at this time. I wouldn't necessarily recommend high speed, but you can most certainly get through it. I'd think the danger of hitting potholes is greater to you (from the resulting fall) then it is to the bike. I wouldn't make a habit of riding off curbs, bunny-hopping, hitting potholes, etc., but the bike can most certainly take it. Get some good wheels, and you're good to go.
As another poster said, look up some pictures from the cobblestone sections of the Paris -- Roubaix race. Greg LeMond once compared it to riding over a miniature graveyard.