If you had to put 10 lbs and carry it in a backpack on your ride, do you think you would notice the difference? Of course you would, so if you had a 10 lbs lighter bike, that would be 10 lbs less you would be hauling around for 20-40 miles. The bike I bought is about 5 lbs lighter than the bike I was going to buy, and the first time I rode a 5 lbs lighter bike I could immediately tell the difference.
If you can find a nice lighter bike you can afford, buy it. If you can find an even lighter one you can barely afford, go with that one instead. If I had to do it over I'd have gotten one even lighter, but that will have to wait for my next one.