I think every one has their own personal limits to what's a comfortable descent. But I also think we can expand that range over time.

I used to be hugely afraid of descents. An especially bad characteristics when you live in a hilly area. Only way I got over it was slowly learning to trust myself, my bike and going down known routes. That built confidence to tackle other things. Eventually, I was cruising down mountains at 60 mph and navigating switchbacks confidently.

It just takes time. As some others have suggested, I'd say balance what your comfort zone is with maybe dipping that toe in the water a bit and seeing where it goes. Heck, it might even turn out to be fun.