It's all marketing and tradition. It really doesn't matter how the frame is shaped, as long as it carries the loads propery and puts your seat, hands, and feet in the right places.

Road bikes probably have a level top tube because it is easier to size and build a frame using a level tube as a frame of reference. Otherwise you'd have to do a lot of trig to get the proper angles to build the tooling to hold the tubes together as you brazed them and if you build the tooling yourself, it is more likely to be inaccurate. Fit and handling flow from the frame geometry, so it is best to get these right. Of course, nobody but custom bicycle builders build their tooling by hand anymore, so this reasoning is irrelevent. Incidentally, compact frames have gotten much more popular because it is easier to make the frame laterally stiff and light using the sloping tube geometry. Looks ugly though, IMHO.

Mtn bikes went a different route. Geometry and sizing are less important because everything is always changing anyway. Mtn bikers regularly change their seat height during a ride depending on the terrain. Road cyclists seldom do. However, rigidity is very important, so they figured out long ago that sloped tube frames are more rigid while being lighter than their level tube cousins. And then there is the standover height issue too.