Try and find a comparable frame to ride and see if you can ride it. If there is a great difference in the ride and you like it way more, than do it.
Personally I have a CF bike and a 45 year old Italian steel bike, I ride the steel and use the CF bike as back up. I just like the steel better but I'm certainly in the minority.