As I age, I too worry about crashing and having long recovery times or permanent injuries. I spend a lot of time analyzing my ride as I ride. I also do this when I drive over the same roads. Looking for blind spots, seeing where I am not paying attention, etc and I feel this helps me improve my defensive cycling skills.

You seem to ride a lot and I also question your judgement to speed up with signs of miserable road conditions. This is not the decision an experienced rider would make. I think you should look for safety courses in your area or find and read online safety sites.

I have no dependents and I don't care if I die in a crash, but I do worry about major disability, brain injury or vegetative state. If I were to become afraid while riding the bike, I would stop riding. I don't know what your family situation is, but I think that discussions with them will help everyone.

Should you stop riding? Only if your doctor cites medical reasons for doing so.

Perhaps you should move this post to the 50+ forum.