Don't force yourself to go faster than you feel comfortable. Also keep the cardinal rule in mind -- never go faster than you are willing to crash because someday you will.

Don't feel obligated to speed up just because everyone is blowing by you. Bombing down hills is all about relaxing, and if you're not ready, you just need to wait. If something goes wrong, maintaining a cool head is the best way to avoid/reduce injury.

In all likelihood, the fear will go away with time as you become more experienced and gain confidence. There's no need to rush things. As you become a better rider, you'll get better at handling difficult situations and you'll also be able to control your bike better in a crash.