How much do you weigh and how much air do you run in your tires?
As far as frame material. I went from aluminum to Columbus steel and the ride is noticeably better. My Steel Wabi weighs just under 19 lbs and is every bit as spirited a ride as my aluminum bike was. I don't ride gravel with it but I do hit a tough section of chip seal and it's not a problem.