I love how these threads always degenerate into some grand inquisition with the goal of blaming the OP for the mishap.

Y'all weren't there, and there isn't a black box to minutely pick apart the details.
Oblivious guy wandering further and further into your lane is not a one-off problem--it's possible to discuss how you handle it without jumping to accuse the OP of doing it wrong.