I ride my Trek Checkpoint (gravel bike) on intermediate single track. Obviously no suspension and it is doable. I have a lot of years of medium to advanced single track riding I my past which helps me ride the Checkpoint on trails. With that being said, I am in the process of getting a mountain bike with suspension for the trails I ride. I'm tired of taking the abuse on my neck and back.

I would recommend, as other here have suggested, going with a front suspension. There will probably be a time when you are going to want to push it. The extra weight is minimal and well worth the trade off. There is one exception, I do have a fat bike that I changed out the front suspension fork for a rigid but I can lower the pressure on my 4" tires (10-15 PSI) to absorb a great deal of shock.