To be fully inclusive, you'd have to include, trans, intersex, and gender non-conforming.
According to Wikipedia:
The prevalence of intersex depends on which definition is used. According to the ISNA definition above, 1 percent of live births exhibit some degree of sexual ambiguity.[8] Between 0.1% and 0.2% of live births are ambiguous enough to become the subject of specialist medical attention, including surgery to assign them to a given sex category (i.e. male or female). According to Fausto-Sterling's definition of intersex,[9] on the other hand, 1.7 percent of human births are intersex.
So the number is small, but not vanishingly small. However, they're invisible because they're made to conform to one gender or the other.
For the record, I am a man, but neither straight nor white.