When building my recent bikes, I weighed each component before installing. My conclusion is that no one thing contributes to the high or low weight. Each thing does in a small way, because, I guess, there are so many components.

If you want the joy of riding a lightweight bike, the two things that contribute the most to the FEEL of a light bike are light rims and light tires.

The hardest thing to change once you've built the bike is the frame, which is one reason to pay attention to frame weight.