Happens on narrow streets around here on occasion. I think the likely explanation is that people guage speed based on their beliefs about how fast a bicycle should be able to go and don't really look at the speedometer.
Some drivers around here, once they try to start passing and realize that to do so would be dangerous for them (particularly on streets lined with parked cars), back off. Others... not so much. Oh well, plan ahead and anticipate what situation you will find yourself in. Plan escape routes.