Actually, headlight optics have been improving ever since the old sealed beam headlights left the market. The best headlights these days (whether HID or halogen) hardly shine above horizontal (service manuals actually recommend aiming them below horizontal) and put out a fat, even spread of light on the ground with little or no hotspots. They do have a bit of spill angled up and to the side of the road (to the right for most of the world, and to the left for the UK, Japan, and other right-hand-drive countries) for better lighting of roadside signs and hazards.

My Dinotte lights up signs better than the headlights of the cars near me, but that's not because it's brighter (it isn't) -- it's because it just sprays light everywhere in a big circle.