Electrik nails it on the head. It feels creepy from a stranger which I have just overtaken. If it's in a group ride or with people I know, I wouldn't think twice.
There's still an element of danger in a commuting route where there are so many variables like potholes, suicide squirrels, cars that pop out of plazas, iZombies (i-whatever pedestrian) etc...
I don't like tailgaters either.
Op, are you sure he was close enough to be drafting or was he just trying to keep up with you?
Maybe I should change my handle to OP someday. I didn't take further look because the feeling was too weird. But the fact that I can hear his clickety-clicks from the rear drivetrain tells me that he is quite close.