I really think that it depends on where you live, and how you ride. I've definitely been blinded and stopped, until the spots cleared from my vision, after passing someone with a new magicshine light.
People that ride out in the countryside, or on trails, NEED those big 400-900 lumen lights. I'd use one, if I were not a city rider.
People in bigger cities need more light just to make sure that the are noticed, though.
People in smaller cities and towns, where I do most of my riding after dark, should have bright, but not TOO bright lights. I used my wife as a non-scientific tester.
I like my lights bright enough to be annoying, but not bright enough to startle people or make my wife see dark spots when I point it directly at her face (made her sit in the car down the street from me to test). What works here is a couple of 160-200 lumen head lights (depending on battery strength). One for the handlebars (pointed at bumper height for most cars - not directly in their face, but not really for looking at the road either), and another 130-180 lumen light strapped to my helmet (to point at the road, and driver's faces when I don't think that they see me).