Bike's got a bit of "give" you can feel it go through a turn...in crit racing, going through a corner at 25+MPH, it's important..that's why I alluded to the feel of the bike. There's not a ton of flex in carbon..it is stiff.
Most carbon frames are really designed to be racing frames. Aluminum is all over the place from hybrids to mountain bikes to touring and heavier duty riding frames...they are a bit heavier, but are more durable and comfortable.
Think of it this way...carbon is like a sport suspension in a car...stiff but very responsive. But not real cushy. So it depends on what you are after. Responsiveness like a racer, or a softer ride for fun cycling.