I never noticed a rear rack making any difference at all but weight on it making a huge difference for this guy who loves to climb and accelerate from lights out of the saddle and throw the bike side to side in "the dance". With weight back there at the end of a lever the length of almost the entire bike and my hands trying ot fight it with a lever only 16" long,it gets old fast. I do that climbing without the dance, holding the bike steady.
My love is LowRider (or equivalent) racks up front with low panniers on them. There I can do the full dance with a lot of weight. Easy on the hands and I can do a full side-to-side rock.