I was afraid of carbon too. As a big guy, it took me years to come around. Now half my bikes are carbon, I've got a set of carbon cranks (power meter) and am considering a CF stem/handlebar set up. We have a giant of a rider here locally. He's broken two aluminum frames but never a carbon. Always at the seat tube.

CF does make different noises though when your chain bounces around... You get use to it.