How do you have the QR bracket mounted on your saddle? I actually have mine all the way against the back end of the rail and mounted diagonally.
Topeak also sells a
special QR bracket called the F22 meant for "racing saddles" which may do the trick for you. It positions the mounting point further back. I'm thinking of trying it out myself. I also have a medium sized aero-wedge which I don't use because it's sort of more room than I need but also because it gets scrunched up when mounted.
Or you might want to try the
F33 bracket which is actually meant for sprung saddles but may work for you too. It positions the pack more vertically but negates use of the forward securing Velcro loop.