Absolutely! For several reasons.
1) With a noisy bike I am constantly pausing, trying to figure out where the noise is coming and thus reducing my speed.
2) Witha noisy bike I am less likely to ride- the ultimate in slowness.
3) Drive train noise = friction = speed reduction.
Not so sure 3 makes much of a difference but no question 1 and 2 do.