Commuted 90% of last summer with a backpack. Hated it! Sweaty, sticky, sore, awful. Never again.
I bought a rack and a large rack bag (and later, a smaller rack bag for when I need to carry less) and I love, love, LOVE the rack and a bag. My bags aren't side-mounted panniers but on-top bags.