Not really, you clearly lose some speed with the knobby tires on black top (road tires are available... oddly); and getting the Pugsley up to speed is slower. However, once you're at speed, it rolls along just fine - it's physics, once in motion, the heavy wheels want to stay in motion (for reference, see the size of the front rotor).

I see a lot of shocked looks when other riders see me keeping pace, humming along with them. Notably, the looks on riders faces on the Almanzo 100 when I caught up to and rode with an echelon of 10-15 riders on typical gravel bikes (~35mm tires).