Has happened once or twice in 8 yrs of riding. Gives me a little motivation to ride harder. I had a conversation after a hard pull with one guy who turned out to be a parent of one of my son's classmates. I don't ride differently when someones on my wheel so it doesn't bother me.

I think it's mostly an issue in people's imagination as it doesn't happen very often and if it does it takes zero effort to strike up a conversation and sort out whether you want someone on your wheel or not.