You're riding too far right.

If someone blows the horn at me, I glance back and if they're still behind me I move left. I'm certainly not going to reward some a-hole by letting them squeeze me off the road.