I think the drivers may think you don't know they are nearby because they are in a car with the windows up and little outside noise. I do not like their practice because it is disconcerting. I use a mirror, but otherwise I stay focused on the road ahead.
I used to wave them by once I had a clear vision ahead until I almost caused an accident and had to come to a quick stop. That said, I look like a Christmas tree with a reflective caution sign on the rear of the bike.