Guy coming from the back was definitely in the wrong and should have called out that he was passing. However, with that said, if you were far enough off of the right hand side that it made it look more attractive to him than going to your left, that might be slightly on you. If he was exiting also, and saw you riding well off of the right side, he may have though that you were continuing straight and passed on the right thinking it was a better choice than passing left and then immediately cutting across your path to exit. His best choice would have been to slow down until after the exit when it would have been clear what your plans were. Either way, a call-out that he was passing would have been nice.

Even when I am out completely by myself, I try to ride as far to the right as is safe, which generally prevents anybody from being able to pass on the right, much less wanting to. About the only time that I will pass on the right on the road is when there is somebody riding very slow and squirrely hugging the left side of the trail or road shoulder. Even then, I would call it out to make sure they knew exactly where I was going to be.