I broke my arm crashing an aluminium mountain bike, and rode it for a few thousand miles afterwards, over rocks and roots and small jumps. That frame (1999 Specialized Rockhopper A1) is perfectly ok.

I don't think the material matters so much as the way it is put together - but if you are planning on breaking a frame in a remote part of the world, steel is better because you'll be able to find someone who can weld it more or less anywhere.