I have sensed this weighty jersey issue also. I have not liked this sensation. And , I just don't like stuff hanging on my back. Whether it be a backpack or bulging jersey pockets.
I found a pretty sleek, aero, vinyl handlebar bag made my Madden. IT has enough capacity to take the overflow, so I don't have to stuff my jersey pockets.