A lot of that is personal judgement... you can wear it all or none. How often you crash, the risk of injury for the type of riding you do, your age and susceptibility to injury (or really long recovery), etc...
Last spill, I landed square on a rock with my knee. So of coarse, now I'm looking into knee pads. Then I'll wear them and never hit my knees... isn't that how it goes? When I rode trails with my moto, I wore it all (knee/elbow/hip pads, back/chest protector, neck brace).
As of today, I'm only wearing a helmet and considering knee pads riding mostly XC. If I had more DH or park stuff, then I'll likely gear up accordingly.