I disagree with the driver.

You aren't holding up traffic if he can go around you. No reason at all to put yourself at risk near the road edge when traffic can go around just fine. Now if cars get stuck behind and have to wait a good 10-20 seconds--then move over or even pull all the way over and let all go by.

It is more unsafe to ride the edge and around a rock or pothole by moving more into the lane than it is to drive a third the way in the lane all the time.