When it stops being fun.

Unless you are one of fewer than I'd say 1,000 riders WORlDWIDE combining genetics, training, commitment, and no small amount of luck to get paid a living salary to ride a bike (aka the "pro peleton", various regional teams, and their Tri and MTB colleagues), you're in it becuase you enjoy it. Age has nothing to do with it. An area small-town paper covered a recent crit; they interviewed a 65 yo who was D.F.L. in his race (maybe a dozen entries). He loved it and couldn't wait for next year's event.