Exactly!! its just a number. How old do you feel (hopefully, being a cyclist, younger than 50)? I'm lucky to be in great shape (5 pounds heavier than high school), and healthy enough to tag along with the racers 30 years younger than me on Thursday night.. most of the way, anyway.
Do yourself a favor, go get a physical, a colonoscopy, etc., and know where you stand.
Reflect on the fact that your perspective on life is profoundly more 'whole' than those in their 40s, 30s, 20s... I don't think I really would want to be back there living needless frustrations, concerns and 'considerations'. Use this as the time to set right the things in your life that are 'untenable' because you are now mature enough to make it happen; to see that it should be so.
Stay as active and fit as you can, it is harder than it used to be, but also grants more peace of mind and satisfaction than when you were 20.
Laugh and love, that's about all we have. And riding of course.