You're paying for name, marketing, r&d, design, customer service/warranty...
A great frame, the best frames...like a Look or Pinarello or whatever...they magically do everything right: light, stiff, comfortable, handles like it's on rails, climbs well, descends well, and looks good doing it all. I've gone through a Trek 2.1 Pilot, Specialized Roubaix Expert, Blue CX6.5, BH Connect, Cervelo P2C and a Look 595. And I've test ridden bikes by Trek, Bianchi, Pinarello, Colnago, De Rosa, a generic carbon frame, and I'm sure a few others.
Every bike felt very unique...and strengths and weaknesses. Among the bikes I own, I could probably tell you what frame I was riding while blindfolded (kids, don't ride blindfolded). The Look frame was like $4K...the entire bike build was somewhere around $8K. Is this bike better than my other bikes? Absolutely. It combines the best qualities of all my bikes into one bike. Is it worth the ridiculous price, relative to my other bikes? Absolutely not. At 2-3 times the cost, I'm not 2-3 times faster...it's not 2-3 times lighter. It's the whole 'law' of diminishing returns thing going on. It's nice if you can afford it.
As far as carbon frames...I have to say...I recently took a short ride on a generic carbon frame that someone locally got off of eBay. The whole build was like $1200...105 group, cheap wheels. It was a surprising good ride! Not overly light...but pretty stiff, comfortable, handled well...really solid. Maybe some people would disagree...but it seems like it's hard to buy a truly bad bike these days. Unless the fit is just totally off, just about anything you get will be good to good-enough.