For off road much of it has to do with tire choices. It took a long while for aggressive 29r tires to be made. Once available, it opened the gates to the 29r being more mainstream.
I do agree, 29 makes a bike very tall. For us we prefer the 29 for open terrain xc riding, anything technical does better on 26. Plus the 26 is less effort to steer.
PK