In general, I support signaling when safe. I understand that sometimes you need that hand on the bar, and sometimes you also need it to brake, especially the left. I'll try to at give a quick signal when I can, and start soon enough that I can keep it up for a while before having to put my hand back. It can be tricky.

In the OP's situation, if I understand it, he was expecting the rapidly approaching car to pass before the OP really had to move to the left, which is understandable. It seems then like it was the motorist who confused the situation by slowing down next to the OP and staying with him, preventing him from moving left when he should have. How could the OP have predicted that? That said, I would have still tried to make eye contact with the driver (perhaps with kind of a "what the heck are you doing" expression) and signalled emphatically before pulling ahead, OR, if a more rational sense prevailed, maybe just waved him on through.

Stop signs especially seem to promote the reptilian "must pass bicycle" response in motorists. When approaching one, unless the car behind me is really going fast, I tend to move out into the center well ahead of time to discourage that, bike lane or no. (Around here, it's mostly no.) And sometimes the stupid motorist tries to pass anyway and ends up stopped way over in the opposing lane!