If you're riding in a straight line and paying attention it shouldn't matter what side someone passes on. Really.
If you're wiggling all over I'm going to wait until you zig one way and then pass on the other side, probably unannounced, because people tend to go in the direction one calls out, and wiggle more when they hear something than they were already.
If you're making it really difficult I'll pull up behind you and say "Hi" then clearly state in a full sentence which side I'm going to pass on.