Sounds like you did the right thing. Usually if I'm not wanting to stop and chat, I usually just cup my right hand around my right ear to make it seem like I couldn't hear what he said.