For sports Birth Sex or Assigned Sex (defined in your link) is where a lot of the conflict comes from. What seems to matter for sports is a Y chromosome/or not. Those with low T and a Y don't get to make up the difference in T and compete. Those without Ys and low T should have the same restriction.
This stuff is often not fair anymore than the age restrictions are fair. But if we are to group people for competition that is at least as clear as the age part.