you're going light enough you don't need panniers.
save the weight of the heavy bags and rear rack by
doing the bikepacking thing. that would also solve
your potential short chainstay problem.

you could also try adding a lowrider to the front,
(would you need a steel fork?) with a set of
small panniers. weight up front is not for
everyone, you'll need to try it and see.