OR...you could take it as a complement that you were riding smooth enough to draft.
Or...he might have not wanted to "spook" you by a sudden masculine voice right at your ear (you know what the path walkers do when you sneak and speak.
And you probably scared the poor little booger when you spoke to him with your manly presents.
Honestly I have no problem with someone drafting me.... he/she might have rode to far and are doing anything to make it home. And if the draft too long I slid to the side, brake hard, and say "your turn" as they pass, and slid in behind.... maybe we can both go faster and I make a new friend.
__________________
I can't ride and Frown!