When I used to work for a buddy that raced cars, we had a mantra "speed costs money, how fast do you want to go." Not sure the same holds true for cycling. Weight, or the lack there of, costs money. How light do you want to go? Will you notice a difference? Yes. Climbing will be easier on a lighter bike, but the performance gain is not linear with money spent. I would agree with others on the shoes / pedals / bib shorts end of the equation. Outfit yourself better, get stronger, and maybe think about a new bike further down the line.

I realize I have focused on performance, not pleasure. But if you optimize your contact points on the bike with good shoes, socks, shorts, gloves, etc you will be more comfortable and thus should have more pleasure, and maybe a slight increase in performance.