I race against are 150+ on 23 pound bikes, the fact that mine was 31 pounds seems totally unimportant since I still have a 21 pound advantage.
I'm not a particularly talented mountain biker, but I agree with your assertion that bike weight is only one factor among several in how well a given person performs on the trail. That said, as a small guy myself, I find it easier to handle a light MTB in technical terrain than a heavier one. Climbing is easier, and it responds better to the upper body movements that go along with riding in technical terrain.