Passing can be rude if you:
-Pass too closely for the comfort of the rider you are passing. Give at least 3ft, they may not be racers used to close position.
-Pass and then slow down in front of other rider.
-Pass when the rider you are passing is likely slowing down for a red light/stop ahead.

When I pass I rarely say anything, but I also pass using far left side of shared lane or in the adjacent lane which usually means at least 6ft of passing clearance and in a place where normally there would be motor vehicles passing the cyclist.