If people pass you on the right it's because you left a big hole there, and probably looked like you were going to force them over if they passed on the left.

And never try to get out of the way-- if they're behind you they can see you better than you can see them. Ride a straight line at constant speed and they'll get around.