Yes he must have had baggage. And he wasn't the sharpest knife in the drawer. When I am riding solo, I don't really make a big effort to be consistent and predictable. If someone was drafting me and I did not know it, well they could end up bumping my rear wheel and going down. And it would serve them right if they did. I am aware enough, though that people can not "sneak up" on me like that (or at least not usually). So you should always ask to draft.
The guy you ran into was 1) rude and 2) stupid (at least about drafting). It wouldn't have killed him to ask to draft, take a turn at pulling you and thanked you for your consideration. I like to have as many friends out there as possible.