Pretty much what Steely Dan here is the heart of the issue. Lighter wheel/tire combinations mean more to "performance" than weight anywhere else.

However, depending on the weight and skill of the rider, this may be a dubious place to start shaving weight. Lightweight wheelsets need to be ridden "lightly". That's not to say that you need to be 150lbs. or less, but the skill to ride a bike without pounding through potholes or running over bumps and curbs. When I first started riding a fairly nice bike (about 20 years ago now), I destroyed the rims because I didn't know how to ride, or more accurately, I didn't know how to ride a nice road bike. My previous experience had been on a Schwinn Varsity and a very heavy Nishiki. Both bikes could be ridden/pounded without concern. Point being, unless I'm an experienced rider with a deft touch on the saddle, this is not where I start shaving pounds.

Where I think a lighter bike matters is purely in the enjoyment factor; a 20lb. bike will feel much more lively than a 27 lb. bike, even picking it up to go riding somehow feels better. Something that is more enjoyable is something that a person is more likely to stick with for the long haul. If one continues to ride their bike regularly, the body weight is going to take care of itself.