I did my first BMX race at 56 yrs and loved it (cruiser class), kept it up for one season but there was just no competition at my age. At the starting gate I would look next to me at a 20 yo kid.
Most local tracks aren't so radical and just take it easy. Jumping ain't necessary.
I did win many seconds and thirds though when there was only another racer or two 35 +.
Highly recommend going for it, great for boning up on the out-of-saddle pedaling but do it for the fun, not to kill thy bod.
Forgot to mention that I really sucked at it but sure had fun.