It's very different in that range than it is in the 1.5" to 2", or even 4" tire range.
Having also ridden fat bikes, I hardly think that aerodynamics is the reason why they're so much slower.
We could do hyperbole and hypothetical scenarios all day, but all that matters in the end is the real world results. As several other posters have said (and has been my experience as well), when you're in the 1" to 1.5" to 2" (or bigger) range, it's pretty consistent that going to a small tire produces a speed increase. Whether that speed increase is large enough to make a difference to you, or whether you find the fatter tire more comfortable so it doesn't matter for you, fatter tires in that range roll a little slower than skinnier tires.