Good to hear that it all worked out as designed in the end - even down to refunding your return shipping. Cool...
When he asks you to remove your negative, just ask him: "is my feedback factual? Well, then, why would I remove it?" and be done with that little hassle, too. The point of feedback, after all, is to alert those that come after and you did them a service by highlighting the seller's issues in this particular instance.
DD