You'll probably want to use much lower pressures. In addition to sometimes being uncomfortable over long rides, vibrating the bike isn't a great use of your forward momentum. Tire deformation does cause a loss in performance at lower pressures, but high-end supple tires substantially mitigate this.

Switching to 28mm might make sense, especially if your 25mm tires aren't width-matched to wide aero rims (you have less to lose). Just don't skimp on the tires if you do; switching from a great 25 to a bad 28 is pointless, because you'd have to run the tire stiff anyway to avoid excessive deformation losses. Then you'd have a heavier setup with no benefits.

I was curious about which particular tire models. Because...

...if your .5-1mph deficit was on really gnarly gravel tires, it's possible that with high-performance road tires you'd be just as fast on roads on your "gravel bike" as you are on your road bike. At which point considering racing it wouldn't be all that odd.

It would be nice to have research that targets and quantifies the comparison. It seems obvious that tubulars should do better on rough stuff at a given width than clinchers, but the only direct comparison I've seen (from a Bicycle Quarterly) showed the opposite happening; a tubular which had slightly outperformed a clincher on smooth roads wound up slightly underperforming on rough stuff. BQ was at a loss as to why; perhaps it has to do with losses at the particular rim/glue/tire interface, which has been implicated in having significant performance affects before.