i'm so sick of hearing people talk about how the only reason such-and-such part catches on is because the manufacturers shoved it down our throats. has anyone ever considered that maybe they have our best interests in mind, and that maybe the reason why something becomes so widespread to become the new standard is because people actually like it? years ago, shimano came up with the brilliant idea of making a sort of squar-ish chainring, they called it bio-pace. bike manufacturers spec'd it for a couple years, and then the consumers decided they didn't like it and stopped buying it. and thus, it faded into the obsolecance.
the threadless system is, for the most part, better and people like it, and therefore, it's not going anywhere until someone comes up with something else. it's stronger, it's stiffer, it's lighter, and most importantly, it's safer than threaded systems, especially on mountain and BMX bikes. on a road bike, you still see those same benefits in stiffness, but i'm not entirely convinced that it's necessary. i do just fine with my quill stem and threaded headset. i definitely notice some flex when sprinting but i think it's mostly in my handlebars (salsa pro-road, 40cm). but i digress...
to offer some thoughts on other topics brought up in this thread...
for the height adjustment issue, i can assure you all that someone, somewhere is working on that. since threadless systems debuted, that has always been a sticking point for a lot of people (myself included). however, there isn't a good way to do it now without negating many of the benefits of the system.
one of the main reasons why steerer tubes are cut short on a lot of production bikes is the material of the steerer tube itself. on a CF steerer tube, you can safely only have about 1" of spacers between the top ofthe headset and the bottom of the stem (for insurance reasons). on an aluminum steerer, you can have about 2", which should be enough for most people. you can go higher on a steel steerer tube, but most manufacturers have moved away from that to save weight.
i work in a shop that sells more bikes than i have ever seen a shop sell, to so many different customers with different riding styles and goals, and i can say that the typical threadless setup works for most people. there are always going to be exceptions.
i think an advantage of the compact frame phenomenon is that while it narrows the spectrum of available sizes per model. it streamlines the manufacturing process, which costs the companies less $$$ per bike model, but also allows them to make MORE models with varying geometries for varying purposes. so instead of an industry where there is one type of road bike with one geometry set per company, we have multiple road bike models for multiple purposes. this gets more people out there on bikes and isn't that the whole point?
a lot of companies that make more "comfort" oriented road bikes, (specialized roubaix, cannondale synapse, the litespeed sienna, the raleigh cadents, the trek pilot) the head tubes are taller above the top tube. this puts the stem effectively higher than it would be on a more racerboy bike. the more "comfort" oriented bikes are still very high zoot performance machines, but designed with the idea that not everyone needs or wants to be in a super aero tuck all the time.