Yeah I'm with you on that.

I mostly commute on one of my bikes with racks and I always vastly preferred panniers, but I've added a lightweight road bike which I love to ride and the only option is a backpack if I'm commuting on it.

I bought a backpack designed for cycling, a Deuter Race Exp Air, and it's amazing how much more comfortable it is compared to a standard backpack. The design keeps most of the fabric off my back and allows air flow as I'm riding. It's a world of difference!