Here's an informational page:
http://www.helmets.org/guide.htm
From that page:
Replace any helmet if you crash. Impact crushes some of the foam, although the damage may not be visible. Helmets work so well that you need to examine them for marks or dents to know if you hit. Most manufacturers recommend replacement after five years. We think that depends on usage, and most helmets given reasonable care are good for longer than that. But if your helmet is from the 70's, it's time to replace it. Replace the buckle if it cracks or a piece breaks off. No one requires you to replace your helmet, so give it some individual thought.
I agree with this assessment. My helmet never gets dropped, and it's properly sized so I'm not warping it when I put it on my head or anything, so it's probably good for a long time.
What would be interesting would be to set up the same safety tests that they use for initial testing of the design, and put some 5 year old helmets through it. I'd be willing to bet money that those helmets work every bit as well as they did when new, if they haven't taken a crash impact. Sure it's a destructive test, but on a helmet I was going to retire anyway I'd do it.