The second statement makes the first obvious.

Air resistance increases as the square of speed. That's a second order linear relationship, not an exponential one.

To the subject at hand: from experience, the trailer will be harder work than the panniers if your bike is built properly.

I used to compete in triathlons and since I wasn't a good swimmer (23:30 for 1500 m) I would spend most of the bike leg overtaking the better swimmers, especially on hills. I rode the the only bike I had at the time which had pannier racks and since I like annoying triathletes, I would occasionally ride races with panniers attached (with empty cardboard boxes in them). I take from thsi that the added air resistance of panniers is quite small.

I rode the same bike with a kiddy trailer for a while (until we found they were illegal where I lived). I never tried competing in the tris with it attached but the drag effect was very obvious.