When you get excessive play in them, and/or pull out unintentionally, I'd replace.

I have about 4000 miles on my current pair, but I only put a foot down at the beginning and end of my rides and don't walk around in them, so they can go for quite a while.