The reason that horizontal persisted for so long was that there was no need or desire to change them. Bicycles. particularly high grade bicycle were steeped in European tradition, until the outsiders ( Japanese and Americans) invaded their turf.

The high grade European bicycles were built on the artisan philosophy. An apprentice serves with a master craftsman, who teaches him the "correct" way to build a bicycle. One did not deviate from the master's method. While this approach results in fine product, it also stifles innovation. Whatever creativity a novice has at the start of his apprenticeship, is often diminished to point of near non-existence by the time they are ready to set up their own shop. Besides, why change something that works perfectly fine.

There is also the matter of sporting regulations, which often discourage innovation or at least limits it. When recumbent bicycles and bicycles with streamlined fairings demonstrated that they could shatter existing records, the UCI promptly banned them. When derailleurs started being popular with racers, Desgranges banned their use in the Tour de France for almost a decade. For a while, he even issued standard bicycles to every competitor, so that no rider gained an advantage due to his equipment. When disc wheels were introduced in the mid-1980s there was a huge outcry to ban them. Whenever, some outside-the-box-thinker, such as Graham Obree, came up with a revolutionary idea, they would often create new rules (or interpret existing rules) to make it illegal.

The net result of the deep tradition and regulations was that innovation was typically incremental. There were exceptions but it was rare. Whenever there was true innovation, it was rarely widely accepted by the consumer, if it had not trickled down from the ranks of pro cycling.

Things started changing with the arrival of the Americans and Japanese at the high end of the sport. They were not mired in the European mindset. They pushed the boundaries, within the rules, and found commercial success, to the point where the Europeans could no longer sit on their laurels, They were forced to innovate or die. The late 1970s started what would be a golden age of innovation in the cycling industry.

One of these Japanese innovations is what led to decline of popularity for the horizontal dropout. Shimano's SIS indexed shifting system proved to be major commercial success and led to many copies. However, correct chain gap is one of the critical parameters for optimum performance of indexed shifting systems. Reports of poor indexing could often be traced to an owner or mechanic who had inadvertently changed the axle position in the dropout, thus altering the chain gap. The move to vertical dropouts mistakeproofed this parameter and drastically reduced indexing related complaints.

As to the origin of horizontal dropouts, I strongly suspect the origin was prompted by the invention of the pneumatic tyre. An axle slot with the opening at the front end makes it easier and quicker to remove and install the wheel. Frequent punctures were an accepted fate on early, unpaved roads, so the advantage of quicker wheel removal and installation was substantial in competition. It became de rigeur when racers started utilizing flip-flop hubs, with a fixed fear on one side for relatively flat terrain and a larger freewheel on the opposite side, for ascents and descents.