For the OP I would just bungee the backpack on the rack.

I have panniers, but to me a backpack plus bungees is more convenient off the bike. I only use the panniers when I have to (carrying lots of stuff or something big).