Opposite for me. I commuted with a bike-specific backpack (ventilation gap & mesh, etc) and didn't like it for anything more than a couple of miles. I particularly disliked it on hills, where I felt like I was riding with a pig on my back, and the heat factor was ridiculous, despite the ventilated design.

I far prefer riding with nothing on my upper body. I ride with a rear rack-mounted pannier (or two) and like the lower weight distribution and the freedom of an unburdened body. The tradeoff is that rear panniers require adjusting one's hill climbing style, especially if accustomed to conventional roadie style bike flicking up a hill (as I was). such flicking sends the back-weighted tail into a horrible dynamic. Instead, any out-of-the-saddle climbing needs to be done relatively smoothly - the bike sway stays on a pretty even keel with little side to side swaying. The rider's body can bob, weave, etc, while out of the saddle but the bike rides steady and solid.
Once I made the necessary adjustments to my perspective and style, panniers became the clear winner for me.

As far as aerodynamics, I really don't think it makes much difference. It seems to me that most reasonable (not HUGE) panniers are essentially tucked in each leg's slipstream, and at very least do not pose much of a new facade to the wind.

If your desire is to increase your overall riding pace, there are many ways (in the category of "training") to do this, and the racing forum might have some training tips and drills.