Ok, I'm old. I started cycling in HS in the late 70's. The only time anyone has bumped my rear wheel has been intentionally to say 'hi' or during a race. Only once in thousands and thousands of miles over nearly three decades of training, paceline riding, in pairs or threes, stranger or friend etc... has anyone ran into my rear wheel and that was because a dog ran in front of us. I didn't go down and no gear was broken. Is there a risk? Sure, but I don't think it's any greater than being hit by lightning.
Having someone on your wheel at 20mph in the big scheme of life is not dangerous. Like others have said, the danger is to the rider doing the drafting. Sneaking up on you? Are you wearing head phones?