Admittedly, he should have waited behind his stop line until safe to proceed, and then proceed.
But then again, it is kind of a gray area.
I have no doubt that the answer in the A&S forum would be to drag the driver out and hang by thumbs and thrash with a cat-o-nine-tails, but it sounds like you were both playing it relatively safe, and the driver felt like making it a big deal...
1/2 point against each of you. How is that for walking the line?
If I were you, I don't know enough to say whether I would have done the same... It would probably depend on how close the ped was to the side of the road. For my safety, if the ped was anywhere other than very near the center of the road, I would have waited behind the car to avoid being sideswiped if the car pulled around the ped.
Bottom line to me, it is a gray area, and you shouldn't worry, but as a cyclist I do tend to err on the side of caution.