Put something heavy on your rack and ride around. I've had stuff on mine with no panniers and it's noticeable, but perfectly manageable. More top heavy than nothing on the rack, but I've never felt particularly unsteady. But then I've never taken a tour that way either, just taken short jaunts. But get some weight up there and see how it feels. I'd rather have my stuff seated lower in panniers, but if that's not what you want to do, see how the bike handles with a comparable load. I doubt it'll be too bad.