I used to ride fast, twenty years ago. I've slowed down.
I think that when a motorist wants to pass, he doesn't care about the speed limit, nor does he care about your safety.
I ride slow and watch my mirror. I go slow enough that I can pull over to the curb.