What's the rule on swerving to avoid monstrous potholes?

The weather was great, so I went for a ride.

Along the path, I saw a deep pothole pretty close in front of me, so I jerked left to avoid it, and the guy behind me, some self-righteous ******* on a Caad 9, yells "hold your line!"

I mean- he wasn't even following that closely. I think I have the high ground on this one..