There are pros and cons. In my opinion, bike handling doesn't improve with panniers, - if anything it's worse, especially at low speeds. Different bikes are going to be impacted to different degrees. That's not to say it's unsafe, unmanageable, or even noticeable, but some posts have implied that bike handling is better because the weight is lower. I haven't found that to be the case at all.
There are advantages to having the weight on your person vs. your bike. You can "unweight" when going over potholes and curbs letting your body act as a suspension.
Then there are the convenience things. On my first day with panniers when I stopped at store for my morning caffeine it was like: "Oh, now I've got to take these off and bring them with me". I carry a laptop and no way was I going to leave that on the bike. Using a basket and throwing a backpack inside it alleviates this problem to a certain extent. Some panniers unhook from the rack pretty easily too.
Then there's the performance issue which does't matter to a lot of people and may not matter to you either. I ride a nice speedy road bike most of the year. Although it handled fine I definitely noticed the panniers when trying to accelerate or ride into a headwind. There was a difference climbing hills too. For me it took some of the fun out of commuting.
Now, a lot of people would be much more bothered by the extra weight on their back vs on their bike. I can understand that and if my commute were 20 miles instead of 6 I might want the load off my back too. I also don't give a crap about a sweaty back. It fits right along with my sweaty arms, chest, legs, and head. I shower when I get to work.
Just giving you a different perspective.