I concur with others on going with narrow slicks, preferably at max pressure. I converted an MTB this way and it made a huge difference.
My MTB had higher bars than yours, and so reducing wind resistance would have been my next suggestion, but you may already be as low as you want to go.
By changing forks you might save a little weight, and maybe save a little energy if you are cranking hard, but not much. Ride it on slicks as others have suggested and see how it feels.
It's possible you'll want a larger top gear on roads going downhill. Wait until that actually happens before bothering to do anything about it.