You can try getting the rack back farther, and loading the pannier as far back on the rack as possible.

But really, you're having the problem because the Langster is a track bike, with very short chain stays (390mm). Touring bikes have chain stays in the range of 445mm, allowing room for the racks to be further back, and providing more stability under load.