The people in the car were probably thinking:
"How long will this guy sit back there before he decides to pass us??"

On a road like that, the double yellow line is merely a suggestion.