I don't think there's any way you could NOT notice a 4 pound weight reduction OP. Also, like JoelS said, loosing rotational weight on the wheels is a reduction that you certainly will notice.

I've changed out a lot of stuff on my bike, 3T bars, stem, seat post, new wheels, saddle, skewers, brakes, top cap, bar end caps, etc. While some of those make a noticeable difference in how the bike rides and/or handles, and some are just weight reduction when I went through that stage, the wheels made the most drastic night/day difference for me.