I'll toss in my three cents worth. I'm guessing you mean trail riding - although routes like the Great Divide Trail are mostly dirt road - not trail. I've done considerable off-pavement touring in places as extreme as Alaska, the Yukon, and the NWT and all over the West. I have an 18-year-old aluminum frame Trek 8000 that I modified for touring - drop bars and a Flexi-stem.
Although it's time to get a new bike - I have never had one problem with the aluminum frame and I really put it to some extreme tests under weight - like Schofield Pass in Colorado or Heckman Pass in BC where the rims were smoking. I use high-rider racks in front. I used to have basic Performance panniers that I had to stop and collect every now and then as they would fly off even with extra bungie cords. My Arkels have never done that.
I am a BIG believer in dirt. I have lived in Wyoming for 15 years and know that the really good stuff is off the pavement. The Canadian national parks in the Rockies have old fire roads that you can ride back to remote campsites. There are great forest roads on the BC side of the border - Elk Lakes is superb! Then there are the Dempster and Denali Highways in the North Country. In fact, you have me thinking about a new thread.
Best - J