Yes! Come join us! There are Masters and age group divisions in every discipline, and you will find camaraderie and stiff competition at nearly every race. I started racing at age 50, and I'm still loving it at 60. I've raced Road, CX, TT, and gravel, and am contemplating track next season.

A couple of suggestions:
1) Try a weekly local crit for starters. The races are short, and they should have a beginner's, or C race. Don't worry about being 30 years older than the competition. In your first several races, it's all about learning to race and getting comfortable riding all out in a tight pack. After you've done enough crits to feel comfortable rubbing elbows and cornering at high speed, branch out into other disciplines.
2) Don't make the mistake of assuming masters 50+ races are easier than all ages cat 4 or 5 races. Masters races are chock full of guys who've been racing for decades, are former college/pro racers, and/or are simply very committed with the time and money to get serious. Older racers are generally safer and more predictable than the youngsters, but they are FAST. Go ahead and do the cat 4 events until you're more experienced.
3) Most states offer some version of the Senior Olympics. These are short road races and time trials, and attract a wide array of riders ranging from casual group ride aficionados to the aforementioned Masters racers. You can quickly find other racers at your level and find whatever level of competition you are looking for.
4) Most of all, just get out there and do it. Racing is a blast, and there is no preparation for racing better than racing.

See you at the races.