Ever since the Rover Safety Bicycle, wheel diameter has been separated from gear size, so it is really an issue of what size will fit the frame. You use the largest size possible to reduce rolling resistance. the limit being the length of the head tube and toe clip overlap
The 700c/27" size is the largest wheel that can be fitted into a normal size frame. There has been some rationalisation, so all the various 27" sizes are now reduced to a standard 700c .
700c does not work in the smaller sizes of road frame, hence the need for 26" wheels (650c).
With MTBs, the 26" was a historical accident. I dont see how it can be the ideal wheel size for every off-road rider. A 6'6" rider could use fat tyre 700c (29"), and a very small rider could use 24" and they would both have a wheel which sits happily in their frame with plenty of clearance. Small kids use 26" MTbs off road, and they are using big wheels for their size.