I have one carbon fiber bike and two steel bikes that I ride regularly. The cf bike is ten years old, the steel bikes are twenty-three and thirty-four-years old. I bought the cf frame used, an ex demo in pristine condition, mainly out of curiosity. It does dampen road buzz, but it is also jarringly stiff on seems in the pavement or potholes. All in all comfort levels seem to even out. More road buzz on steel, but they are springier on rough roads. Being stiffer and about four pounds lighter than either of the steel bikes I can average slightly faster speeds on the cf. That being said, on mostly flat roads my fastest bike is steel.
The only aluminum bike I've ever had was a mid-range Raleigh cyclocross bike that never quite felt right, but that was due as much to geometry as to frame material. I was using mostly as a foul weather bike on the road. As a cx bike it probably would have been fine.
In short, it's all good. Fit and fitness are what really count.