My suspicion is that drivers believe you are signaling a movement to your left, and it makes them wary. If you ride frequently in the same places they'll probably eventually realize that you're not signalling and you won't perceive any benefit at that point.