I'd first tell him to stop being so damned critical of you because being critical of someone else is too easy. Tell him to be critical of himself- he's not perfect so there's something to be critical of.

Anyhow, what I've found is that people who drive and don't ride are very cynical and pessimistic about the abilities and practiced responsibility of other motorists. Remember that these very people are overly sheltered, though; their mode of transportation has been that of their cages and they've become conditioned to feel naked outside of them. To put it another way.. if you wore a shield of armor every time you stepped outside you front door and derived a sense of security of having it on, you'd feel insecure at the thought of not having it. IOW these people are insecure. Moreover, most if not all of these people haven't even tried road cycling. They don't know how motorists react to you when you're on a bike or what it's like to ride in traffic. You could tell your friend this but in his self-righteousness I doubt he'd actually consider any of it.

I fortunately don't have to endure comments like these but if I had to hear them, I'd tell first tell the other person to mind their own damn business and to stop being so lecturesome. If he persisted, I'd tell him something like what I said above.

Regarding your doubt, I don't think it's a bad thing if it forces you to ride more cautiously than before. I have developed the habit of catching myself when I become too comfortable, telling myself to be a little nervous and more alert. Regarding the risk we take, yes, it's real and any of us could be killed in an instant. All we can do is to minimize the risk, through our own behavior, and then to accept it. Or stop riding. Take your pick- doing what you love and living how you choose or not taking the risk and having a higher degree of safety. Can't have both, really.