I used to spend a lot of time riding at night on rural, winding roads. I found that at about 1000 lumens, I no longer had to go slower than I normally would in daylight when descending hills, but having more like 2000 lumens felt a lot better. A car headlight is around 1300 lumens and they have two of them.

But more importantly, having car-bright headlights and very bright taillights (100+ lumens) got me substantially more respect from drivers. They would pass me with a much wider margin after slowing down. Coming at me, especially at corners, I was much less likely to have trouble (left or right cross issues). The fact of the matter, drivers are not expecting to see a cyclist at night and it scares the crap out of them when they come across a cyclist they did not see. Powerful lights help them see you well in advance with no surprises for anyone. So my advice would be to invest in appropriate lighting that makes you look like all the other vehicles on the road.

Lumens only tell part of the story but most bike lights through enough of a directed beam downrange that it's a decent relative figure of merit.

As far as beam shape goes - I go with the primary concept that my safety trumps most other concerns and certainly over the irritation of other drivers. I'll try to not shine my light in their eyes but my primary concern is being able to see road defects and to be seen by approaching drivers. As a cyclist, I'm the one that is in the most vulnerable situation with the greatest consequences for a problem and I think that buys me some, if not considerable, latitude.