I had a similar experience on a group ride where the guy waved me on after about 2 minutes of drafting him.
I guess I still have a lot of group riding etiquette to learn.
I’m with you Yo-, I could care less who drafts me. But you have to think of the people who are a little uptight.