I'm guessing (hoping) you have a frame mounted rack and not a seat post mounted rack.
A trunk bag put all the weight up higher and the bike will sway and be harder to control.
Panniers will put the weight much lower on the bike so you'll have a lower center of gravity and easier control of the bike.
__________________
The Lord is merciful and gracious, slow to anger and abounding in steadfast love. - Psalm 103:8
I am a cyclist. I am not the fastest or the fittest. But I will get to where I'm going with a smile on my face.