Some fuel to add to this thread.
- Level top tubes due to lugs for the most part. You would have had to stock a ton more lugs with various angles if top tubes sloped since it would have been a different slope for each frame size.
- Sloping top tubes help increase stack height without resorting to questionable kludges and to be quite frank- many people on here could benefit from an increased stack height on their road bikes. I consider myself to be included in that slight insult.
- Threadless headsets solved a dangerous issue in mountain biking and there is no downside to them so they caught on.
- Frames come in tons of sizes, even with sloping top tubes. That was true in the 90s, true in the 00s, true in the 10s, and true in the 20s. Im sure one could find bike shop brands with fewer sizes at various points in the last 3 decades, but I sure can find a lot of brands and models with more sizes now than 30-40 years ago.