Racers use extremely small tires because of their cornering profile (smaller tire = less deflection = fewer wipeouts). It has nothing to do with speed. Wider tires tend to be a bit slower because they have lower pressure limits, but that's a pretty small impact.

That said, I'm just really not convinced on the comfort benefits. I just switched my (hard frame) MTB from a 26x2.125 to a 26x1.75, and am finding the ride much more comfortable. Of course, there's a lot of variables that go into comfort, so it's hard to do a side by side comparison, but I'm just not convinced that size is the main determinant of comfort.