I've been an avid (I ride a bike to escape "serious") cyclist at least since early 70's.
I'll throw out my guess on what's happened. Anything in the way of a road bike in the old days was either a lugged steel frame or a welded steel frame. The botique frame makers were Italians that were making small numbers of lugged frames by hand. Though I've read that they did have techniques for adding a bit of assembly line efficiency and speed. In the US the focus was on selling lots of bikes. Especially to those under the age of 14 or so. In our culture grown-ups didn't ride bikes. About the oldest bicycle rider you'd run across would be a college age student. And that persisted only until they graduated and dumped the bike. Our bikes were only intended to be short term toys or temporary transportation. In that context it's easy to see why the independent frame builder, that considers his product to be as much art as function, can hardly exist in the US. Considering the environment, I believe that companies such as Trek, Specialized, and Cannondale have done a good job at introducing a better quality frame to people that wanted more than a toy to ride. Also, with the introduction of materials other than steel, mass production techniques can be fully leveraged and still produce bicycles that are light enough to cary more than $1000 price tags. In my garage I have an early Trek made with Reynolds 531 and lugged. I also have an Italian lugged steel bike from the same general era. The Trek is pretty decently made, but when compared to the Italian, the production shortcuts become obvious.