Speaking just for myself - for the distance that I travel, the cargo carrying capacity and no sweaty back, I am rack and panniers all the way. My bike was designed to accommodate them, so for me I don't sense any degradation in the bike's handling with them on.