If you're out trying to better your own personal limit for performance (average speed, distance, ascent, whatever) then weight becomes more of a factor.
Making it easier on yourself(ie making the bike weigh less) is only going to slow your progress.. If you want to get better faster, every pound you lose, add back to your bike.