I used to have a tail trunk, which was OK for going to work and back but it was kind of a pain to install/remove each time I locked up my bike. Plus, it's kind of awkward to carry around and I imagine a pannier would be too, unless it had backpack straps. I now use a backpack, even though I ride in 100F weather, at least on the way home. I'm going to sweat through my shirt regardless of whether I'm wearing a backpack or not. But this way, I lose the rack and the bag and it's more convenient when I lock up the bike.

The reduced weight and better aerodynamics definitely made a difference, but maybe only 1-2mph. I've gained more than that by just riding every day and pushing myself to longer, harder rides. Just my 2 cents, I think it's all about whatever works for you.