The point about failure mode is key. If the matrix (e.g the glue) is protected (painted) from UV and not degraded, and the fibers themselves are not nicked or cut, CF has a much higher fatigue limit than steel. But a nick or gash that cuts some fibers is going to be a big stress concentrator. Or hitting a bump and getting some delamination inside the fork may create an unsafe fork.

I guess if it's not nicked or cut, and the "coin test" gives bright, clear clicks all round, I'd continue to ride it. But if you're worried about it, that makes cycling less enjoyable. That would involve a personal choice to replace.