As part of their design, aluminum frames are generally "overbuilt;" this is what gives them their inherently stiff ride quality. I've beaten the heck out of enough aluminum mountain bike frames, without problems, to know that a quality aluminum frame is plenty strong and durable enough for anything that I'm likely to throw at it. Regardless, the big thing with aluminum frames is knowing that since aluminum doesn't have a fatigue limit, any and all stresses an aluminum frame encounters puts it a little closer to failure. Steel has a fatigue limit, so at least some of the stresses a steel frame encounters do not put it closer to failure. That being said, steel frames can fatigue and fail, too.

FWIW, I've bought a bunch of bikes in my life, but never a used aluminum-framed one. To each their own-