I ride mostly on paved roads with my Specialized Diverge with 700x38c tires and 2x 48-34 chainrings. The bike is excellent on paved roads. I also have a Cervelo Calendonia with 700x30 tires. For road use, I ride these bikes pretty much interchangeably. The Diverge is just as enjoyable on the road as the Caledonia. I (almost) don't see any reason to get a second wheel set and tires (although I do have a second wheel set, on a second bike). If I'm riding with others, I'll take the Caledonia because it's a bit faster and more responsive. If I'm riding off road, I'll take the Diverge. But if I'm just doing a solo ride on pavement, both bikes are equally enjoyable, and I'll often choose the Diverge over the Caledonia for a solo road ride.
I'm riding Schwalbe Marathon Plus 700x38 tires on the Diverge, which I got because I was having a problem with lots of flats with the bike's original tires. I like them, but some think they have a harsh ride.
The bottom line on your question is, as far as I'm concerned, a gravel bike with 700x38 tires makes a great road bike, and you don't need a second wheel set, especially if you're riding a mixture of pavement and gravel on the same ride.