Bell and Giro both recommend you replace their helmets every 3 years, but don't specify why. My guess is this is part CYA and part marketing.

As far as I know, styrofoam and plastic are both quite stable. As long as you haven't bashed the helmet, it should be fine for much longer than 3 years.