The rider pulling controls the situation. They signal and move to the left when they are done, absent some other protocol. The other guy could have thought you were rude but more likely realized there is a reason drafting random strangers is bad form.