i definitely empathize with this, and must admit that in my younger days (and still occasionally) it did sting to get passed. especially by those who "didn't look like they should be passing me" whatever that means. i now know that some of the baddest fastest riders out there ride in flip flops and cutoffs and only turn it on when they want to.
i never draft unless i'm training with people i trust, and don't generally mind being drafted as long as the person seems to know what they're doing and doesn't cross my wheel. i say without an ounce of conceit that if someone doesn't know how to draft safely, they probably cant hang on my rear wheel anyway