You were in the wrong because you didn't do what some guy driving a car wanted you to do. People in cars get to tell people on bikes what to do, and they get to express their displeasure at us if we don't do what they want. They even get to kill us if they fee like it. Don't you know these rules? I thought everybody did.
Seriously, though, if some guy in a car is trying to get me to do something I don't want to do, I usually have to resort to grand sweeping gestures of the "no YOU first, I insist"-type. Sometimes I even have to stop and dismount in some circumstances.
I think he was just trying to go out of his way not to right-hook you, a thing that once you get off your bike and get back into your car you find incredibly easy to do, even when the bike rider is riding VC.