Safe riding distance (similar to a car) is that in which you can quick stop directly behind the driver in front of you without hitting them or having to swerve. The flaming you're receiving is because a quick stop when drafting is impossible and causes potential danger to the rider in front, yourself, and possibly anyone on either side.

There was a thread like this in the commuting forum. I've been drafted a few times unannounced and I admit it's frustrating if I notice them my peripheral and know they're right on my ass without them saying anything. The main issue is if a pedestrian or car pulls out and someone is right on your rear then there's a higher chance of being rear-ended. If you've ridden someone's ass and nothing happened then good for you, but understand it's still inconsiderate and a potentially dangerous situation for both yourself and the rider in front of you. Keep in mind track and road riding is totally different, no sudden obstacles and left turns only on a track.

I have been drafted and drafted other riders after a friendly cue and make sure to signal appropriately when there are obstacles, passing, etc., it's just about communication.