Might as well enjoy the ride...
If I ride 15 miles I don't have much post ride fatigue. If I ride 25 miles in high humidity and temp., I may feel drained as you describe, usually after shower and meal. I'm only 64, and just consider it a normal effect of aging. If I ride every day, I try not to go over 20 miles. 16 or so is better for every day of the week riding.
I say just live within your limits and stay active. There's more to life than cycling; don't burn yourself out just to meet some notion of how far you think you should go.