I have a bag big enough to carry what I need for work, but small enough to fit in my pannier. If needed I have a 2nd plastic grocery bag. My empty panniers stay on the bike. Works well for me.

But I agree the backpack gives you more freedom. If I did it again, I might use the panniers for tools and spare parts and put a basket on the rack and put a backpack in there.