Weight of the bike is the last thing that's going to impact me because I'm a built like a whale with arms and legs. It isn't pretty once that all gets squeezed into lycra cycling attire, trust me. Still, I suggested wheels and pedals as a place to save weight rather than other components because as I remember my physics class (long time ago so I'm prepared to be wrong) the weight's distance from the centerline(axle) increased the energy requirement by the square of the distance. Changing the weight at the location only adjusts the energy proportinally though, so it might all be a wash. Still, it is not a net zero impact because of the need to brake as mentioned by others. Pedal weight has to be lifted every revolution of the pedals, so the weight difference again adds up. I think the thing to remember here is that while all that is true, the rider is by far the biggest influence on the performance of the bike, followed by aerodynamic position (also mostly determined by the rider and geometry, not weight) of the bike. Put a peleton rider on a hybrid and they'd still destroy the typical commuter even if they were on a time trial bike with all the goodies.

Long winded way of saying, the math is irrelevant to most of us, use what feels good to you, and don't worry about justifying it.