How much weight do you need to remove from the bike to actually notice a difference?

I have never ridden a 14lb bike, but I'm pretty sure I would notice a significant difference from my current 24lb bike, but would a 20lb bike be a noticable difference? At what point would someone notice a loss in bike weight?