I am not sure how anybody could be ok with someone riding up on them and sitting there unannounced. No matter how long you have been riding or how experienced you are there, so many things can go wrong so fast. Here are two somewhat similar scenarios that could easily happen to all of us.
1. Let's say you are riding on the MUP and someone comes up behind you unannounced. There is a stick/turtle/sinkhole/whatever in your path and you slow down/swerve to avoid it. At the same moment there is a rider approaching you from the other direction. Since you did not signal anything the rider behind you swerves to go around you and smashes into the oncoming rider. Or they smash into you because they saw the oncoming rider. Or they smash into you and cause you to swerve into the oncoming rider.
2. Let's say you are riding on the road and someone comes up behind you unannounced. There is a bumper/roadkill/pothole in your path and you swerve to avoid it. Now this person also swerves to avoid hitting you and at the same time a car is coming up from behind. Oh no! They were not giving you 3 feet. The stealth rider just got run over by a car. Or the car swerves to avoid him, over corrects and hits both of you. Or maybe you tap the brakes and swerve and they run into your back and push you into oncoming traffic.
All of these things are very real possibilities. If you don't think so or still think it's ok for you/someone else to stealth draft we will be looking for your "I got hit by a car" or "Someone drafted me unannounced and I wrecked" threads.