my guess is that if you were carrying 5 lbs or less of gear (maybe a jacket, a spare tube and tire irons, a cellphone, wallet and keys) then it would make little difference for mileage of less than 15 miles without much elevation gain or wind.

For carrying more weight, longer distances and considerable climbing a pannier would probably be not only more comfortable but less fatiguing than a backpack, which would translate into a faster time for the same distance.

Why not do a test? Put ten pounds of weight in a backpack and do a 20 mile ride as fast as you can. A few days later do the same ride under similar conditions with the same 10 lbs in panniers, again as fast you can.