Ive taken road bikes on tracks and trails, curbs, potholes etc. They don't falll apart if you avoid ultralight weight-weenie type equipment. Did you see in the TdF when Lance avoided a crash, and cut a corner across a ploughed field at fairly high speed?
With curbs the technique described above is good, where you pop the front wheel up, then brake, then get the rear wheel over. Riding off curbs is no problem, unless you drop into a rising camber at slow speed. With potholes, you can lift the front wheel over the hole. It all comes down to having a responsive and sensitive technique.
Taking a road bike off-road will improve your riding skills.