I don't think we have noticed a big difference in the frame bags or panniers. We have tended to use the panniers on road touring and the bags on gravel /mixed type of trips. They both suck on big crosswinds. We actually have more issues when running our deep carbon rims on a light bike. We also tend to ride flat Jones bars on the mixed rides and drop bars on the road so that may play a factor in the handling as the flat bars have much more leverage and more upright body positions. Plus bigger brakes as a another benefit on a loaded bike.