The OP is not alone
I decided to try something new today. I mounted my Nashbar Townies to the bike. Put my backpack in the left one and my bulky jacket in the one on the right. Probably 15 pounds between the two... I noticed how sluggish the acceleration was, how it wasn't as agile. I'm going back to my mess bag until the temps climb and then can make the transition to a trunk bag.