There is even a guy here on BF who will phrase it something like "On your left, ladies..." when he passes a paceline of guys. He keeps score on his rides of how many people he passes. Sigh. But it's all in how you say it. I'm a very polite rider, and tend to avoid things that can be construed as insulting. Looking back at someone after you pass them can do the same thing, so I don't look back.