I hate it when someone starts to draft without announcing themselves. As the OP said, there are things like potholes and critters out there. I might swerve without thinking if I don't know someone is back there. Of course I'll probably take them out and be OK myself.

Just last week someone latched onto my wheel and I only noticed because I saw their shadow. Kind of scared me. I always announce myself when approaching another rider if passing or hoping to catch a draft.