It's mostly a matter of weight and geometry. The 120mm fork on my XC bike performs well enough for its intended riding terrain and keeps the weight to a minimum. The 160mm fork on my all-mountain bike weighs over a pound more.

As far as geometry, looking across the spectrum from XC to DH...the wheelbase gets longer, head tube angle gets slacker, front end sits higher, etc. There is a happy medium for each discipline in regards to geometry. I "over-forked" my all-mountain bike from the stock 150 to 160mm travel to make it sit even higher in the front. The "slack" geometry on my all-mountain bike works great for riding extremely technical terrain, however I wouldn't want to race an XC race on it and I even find it overkill on moderately technical terrain. The same could be said in reverse of my XC bike.