It all makes a difference.

I can tell when I switch between bikes that are 7 lbs different in weight on a ride. Especially on hills.

I can't tell the difference in a ride when I've gained or lost 7lbs myself.

I often wonder if the folks that say they can't tell a difference are tracking their numbers. I'm a numbers guy. I have a top of the line cycling computer and a power meter and upload all the data to Garmin, Strava and Training Peaks. I can get pretty granular with it and certainly tell the difference in the numbers between bikes.

I suspect if someone were tracking to that level and went out and rode the same route on two different bikes that weighed significantly different, they would also be able to tell the difference once they see the hard data. I've done that one a number of occasions on a few different bikes.

I'm betting the difference is that most folks just subconsciously adjust their effort on bikes of different levels of efficiency and then assume that they are the same in terms of performance.