Riding 3 or 4 time per week for a total of 100 miles with only one hard ride is not the problem nor is age. I am quite a bit older and ride much more than that and do not have extreme fatigue nor do others my age.

I have had a couple of incidents like what the OP describes. I think I had Covid in the Spring of 2020 and what he describes matches what I felt, it kicked my butt for months. Really weak and no recovery.

I once went to my Doc and explained that I kept having coughing fits on my bike and my power had declined significantly, FTP from 285 to 220 watts with no change in training. He brushed me off saying the power meter is probably miscalibrated. I told him my coach told me to see an MD. I explained I have three power meters and I confirmed my lousy performance with a stopwatch on hills that I have climbed for decades. That I was a EE with decades of measurement expertise did not persuade him. He looked at me and said, "You are the healthiest person in my practice, get out of here". Eventually, I was waking in the middle of the night with breathing labored and had to sit up to breath. It took 14 months for asthma to be diagnosed. When my asthma is not under control, my power and recovery are both poor. I doubt this is what the OP has but it is a possibility.

If I was OP, I would go to as many different specialists as possible. Age is NOT the issue. Something medical is going on.