I think you were fine. As others have mentioned, if you were in a car they wouldn't be going anywhere anyway. You don't owe them anything because you're on a bike.
I often position myself in various locations to allow people to turn right, but this doesn't feel like one of those situations.