This is speculation but I also think with the invention of threadless headsets. It's cheaper for bicycle manufacturers to move the headtube up instead of adding extra fork steer tube length and spacers. IMHO The reason for that nice flat bar with vintage road bikes is basic geometry. The straighter a triangle the stronger and less flexible it is. But with stiffer materials like aluminum and the aide of computer design they can make the frame even stiffer with a much more offset triangle.

Personally I love the lines of the classic road bike much more then the new ones.