I ride on some older country roads that can get pretty rough in places. I have an aluminum Cervelo and a carbon Cervelo. Anyone could ride these 2 frames for 30 seconds on some of these roads and immediately tell the difference. The aluminum frame is literally a rattletrap compared to the smooth ride of the carbon frame. The difference in comfort is amazing especially on a 3-6 hour ride. The amazing thing is that the carbon frame is also stiffer and more responsive. I can't see any downside to the smoothness of the carbon ride. My old aluminum CAAD9 was exactly the same as the aluminum Cervelo.
I also have a steel Bianchi frame and this frame is much closer to the carbon frame than the aluminum.
I wouldn't buy another aluminum frame unless I was really trying to save money or if I was racing the bike and couldn't afford to replace the carbon frame if I destroyed it in a crash. On the other hand, I have seen plenty of bent aluminum frames that were beyond repair also.