Having more weight on your bike does change how your bike handles. Not having been there, it's hard to know if anything's wrong, but I'd guess it's just not being used to the weight distribution. Moving 20-30 lbs from your back to your bike significantly changes the balance between dead weight (weight that just sits there and falls the way gravity wants it to) and live weight (weight that shifts around to keep the bike upright and holding a line).
It's not harder to ride straight with panniers, just different.
I typically ride with only one pannier, and the only time I've noticed the lack of balance was when I had 2000 sheets of paper on the one side.
Make sure that the pannier is firmly attached top and bottom, so the weight doesn't shift, and other than that, it's just a matter of riding. In a week, you won't even notice they're there.