You will be marginally slower and less comfortable with a smaller tyre.

In about fifty seconds someone will come in and overconfidently claim that NARROWER SIZES = MORE FASTERER, but they're the same kind of idiot that pumps their tyre pressure up as hard as possible and mistake the bike juddering over every crack in the road for the feeling of speed.