There isn't a huge night and day difference on smooth pavement. Rough surfaces require larger tires, larger tires require less pressure, less thought about route selection and are less prone to pinch flats. 23mm tires will force you to be more careful about the line you take, possibly making for a faster ride.