I have a gravel bike. It is perfectly fine as a road bike. I have two wheel-sets. One is for on-road with slick tires, the other is off-road with serious treads.
Although I have run narrow tires (28mm is narrow for me), I'm now using 38mm slicks and 38mm treaded tires (both Compass) on the two wheel sets. I detect no performance hit with wider slick tires on road, and it allows me to take it on modestly challenging gravel roads too.
In other words, a gravel bike does not inflict any significant penalty, and it gives you the advantage of running wider tires if you so choose. You might not even need two sets of tires/wheels, depending upon what kind of gravel you ride.
A gravel bike is a road bike with fewer constraints on your riding.