Until you can find a way to cinch down the bag securely, you might just wrap a bungie around the bag and rack to keep it from flopping around.

I ride somewhat aggressively as well and my panniers --empty or full-- always stay in place, so the two are not mutually exclusive.