I think if your load is very light, then it doesn't matter. Especially when the weather is cool.

In the summer I'll pick pannier over backpack every time. It's cooler and more comfortable.

If you have weight or bulk to haul, the pannier route is probably more comfortable. If you can reduce the weight or bulk, that will always be a good thing as far as the riding goes, at least!