It feels the same if I loaded my pannier with stuff as compare to the same bike unloaded. It does make a heck of a difference when I have to carry it up some stair. My back thanks me for the lighter bike.

What does makes a significant difference is my body mass. If I am able to poop the excess load before a ride, that little bit of weight makes a hell of a difference in my comfort level and I can ride better Haha.