I commute with panniers and have had this happen to me exactly once. I did know she was there right away, and thought about dropping her, but soon realized she was a reasonably competent rider, paid attention to signals, etc., so I let her stay there about 10 miles until I turned off. Incident did not bother me, but if she had done anything whacky or dangerous I would have been irritated.

I don't draft other people except in group rides where people are expecting it.