I'm not talking about if you had an accident; I'm just wondering how long think a helmet can go before it "wears out" from normal use.

I was at my local LBS the other day asking if they had a replacement for the velcro and padding that goes in the front of the helmet. He said they didn't, and that I should replace the helmet anyway if it was older than 3 years. Does the padding inside the helmet really degrade that fast? It looks fine to me.