When a 700c rim with 700x23 tires starts to slowly go out of true, what would generally be considered the tolerance that you would let it go before having to get it properly trued again?

Both my front and rear wheels are about 1mm out of true each, and the rear is about 1mm out of round. You can't feel it when you ride, no brake rubbing, nor is there any shimmy or wobble, even at high speed. I am just curious as to how far you generally let it go?