I went for a walk the other night. and was much surprised to heard the phrase "on your left" behind me. What I found interesting is that my body went left even as my brain said go right. It was if my instinctive physical response overrode or rode over(weak pun intended) what I knew to be the correct move to make. Weird!
How far behind you was the rider when he or she called out? Typically riders who voice announce do so as they are passing. Normal reaction is to turn in the direction of the sound which is then alongside you. Had he or she announced further back you would have reacted differently.
The trouble with voice warnings is that a rider who wants to provide sufficient warning must raise their voice significantly, and since that is perceived as rude yelling, many are loath to do it and instead rely on a friendly quiet warning far to close to be effective. I use a bell because it can be heard from a distance and is recognisable to most, though I have been yelled at for not calling out "on your left" in addition to the bell.