I'm with OP, I never let them by and just ignore them if they honk or ask to get by. Once I let a landscaping truck/trailer pass me on the right and just missed getting clipped by the extended mirror. It's a safety issue not a politeness issue.