I ride my time trial bike a lot and I primarily use it for efforts versus just riding so the speed is going to vary against a schedule. If someone gets on my wheel when I am doing a very hard effort, I am going to go really fast but then abruptly come out of the aerobars gasping for air. I do not hit the brakes but the aero drag increases precipitously and the speed decreases pretty fast once I am on the bull horns. I would never agree to allow someone to draft me under those conditions. From time to time, someone will hop on my wheel and I yell at them to get off if I know they are there.
I would not draft another rider that I did not know who was in the time trial position. Their forward visibility may be compromised due to the riding position and one does not have the same level of control in the aerobars that one has in the drop bars. I only ride a TT bike on a course that I know very well and is suited for a TT bike or I ride at the track.