90% of the time I put backpack on the rear rack. More aero than panniers, as well as more convenient when off the bike.

I carry panniers when I need more luggage space, or when it's heavily raining, since they're also waterproof.

Backpack on the back - heavy, sweaty - all wrong IMO.