You should certainly ride what you feel safe on, but I havent seen any reason in this thread or anywhere else to feel that any material is more or less likely to fail. I will say that your estimation of risk is almost certainly not accurate from a quantitative standpoint. Risk in this case is simply a function of number of items sold and incidences of catastrophic failure. Given shear numbers of items available in market, the risk that you will come to own a component that fails catastrophically is without doubt very low (I would guess some fraction less than a 1/10 of 1% of components will fail but thats a guess). That said, risk is certainly not zero, and any part of risk calculus is always the potential impact of failure even if risk is low; and indeed potential impact of a failed steerer or fork is high. So if you fear it, you won't feel comfortable riding so wont ride. So I would not presume to try and convince you to ride carbon stuff. Hell, I tend to steer clear of carbon components that require other components be clamped to them (seatposts and steerers) because I fear I will damage them by over-tightening clamps -- but thats more a reflection of my lack of confidence in my competence than in the component itself.

Frankly, my anecdotal experience suggests that its just as plausible that we are at greater risk from metal failure than from carbon failure. I have had various pieces of bikes fail -- a steel chain stay on a Jamis frame cracked at weld (Jamis gave me warranty replacement frame), a Trek alu mountain bike frame cracked in middle of the downtube (I was not original owner so no warranty), and I had alu flange of a shimano hub crack. So metal stuff from big, reliable makers fails. So far I have not had any carbon components fail, but I certainly believe they do too, but for me, at a lower rate than carbon stuff . My anecdotal experience does not valid data make so is no more meaningful from a data analysis standpoint that your concerns about carbon, and at end of day you could be right about your carbon concerns.

I will lastly note that I actually take comfort in recalls I am aware of (Felt, Specialized) rather than view them as indicator of greater likelihood of failure; to me voluntary recalls mean companies are paying attention to quality (and recalls are cheaper than risk of litigation). I suspect recalls are based on the company's quantitative assessment that some specific lots of their production runs don't meet whatever failure standard they have set. So they are taking steps to ensure things get pulled at early sign of any issue even if the chance that an owner of that specific component actually experiences a failure remains very low -- albeit higher than for components that meet the company standard.