In a windy RR I did this past weekend I was blown off a wheel (40mph gusts) and when I attempted to move back in a guy came up and took it from me. He put his hand on my hip and nudged me a bit. I could have done one of two things: 1) continue to take the wheel, I was in position to do so as I was in front of him or 2) I could yield. I'm a 5. I did the stupid thing. I sucked wind and blew up a while later.
I saw him doing the same thing to another rider and I think he was using it more defensively as he was not "pushing." People were all over the damn road that day, anyway.