Here's what I think happens. Once the car gets far enough past you that you are behind the driver's head, in their opinion you've left their consideration because bikes move so slow.

Car drivers only look in front of them, and have no memory. If they can't see you, right in front of them, right now, you are not considered.