Why not try a set of flared gravel/adventure bars and raise the bars for more confidence on the downhill stuff? I do a lot of gravel and MTB trail riding on my gravel bike and found that I like a lot less saddle to bar drop than I would use on a road bike. My bars are about 30mm higher than my road bike and have a 12 degree flare, which puts your hands/wrists in a more comfortable position for braking from the drops.

Most of the challenge of a gravel bike is riding off road terrain with drop bars. If you switch to flat/riser bars, you just have a rigid mountain bike with skinny tires.