When you say your friends see your tail light perfectly, it makes me wonder what distance the tail light is being seen from. If it's less than the distance between two telephone poles then that is not a good distance to judge a tail light from. Try having your friends stand three telephone poles from your bike and see if they still think it is a highly visible light. I mention this distance because when a car is approaching you at a speed of 40mph the distance I mention is how long the car will travel between the light being seen and the car altering it's course to miss hitting you. It doesn't seem like it would cover that distance in the time it would take for a driver to identify a light, tell himself to change course and then actually make the car do what he wants. But it does.

I like a 600 lumen head light for the type of night riding you describe. I also want a strong wide beam of light (I want my whole side of the road lit), strong so I don't out run it when I go fast and wide so I can see what might be right in front of me and also to either side of that so I can choose a good path around obstacles.

Good luck and be safe.