I replace helmets when they need it. During the time in my life when I rode more than 10K miles per year, sweat, UV, and compression seem to mess things up after a couple years.

I've had my current helmet for two years but it still has quite a bit of life left in it. If you even question your helmet, it's not a bad idea to replace it. They are critical safety equipment and a very minor part of the cost of cycling.