I agree with 531phile. I use panniers every day of the week.

But there are lots of people around who carry everything in backpacks. I've done it a few times myself. I didn't have much stuff and it was OK. Probably warmer in the winter too.

Maybe you should try both and see what works for you.