OP, if panniers make handling wiggly/unstable, maybe look into how much flex is in your load and load attachment. Does/did your rack flex sideways under load, are your panniers or contents shifting around much? Did your rear triangle flex under load? How heavy did you pack?

Counteracting this might reduce your problems while not having to go the bikepacking route on a small frame all the way.