If you're an elite athlete with half a percent body fat weight, climbing up the side of a mountain in a Grand Tour, then saving 4 pounds on a carbon frame could mean a lot.
Personally, I am carrying a lot more than 4 pounds of excess weight that has nothing to do with my bike.