As you gain confidence in the bike, and yourself, you will conquer fear to a whatever extent is comfortable for you. As for me, I have virtually no interest in going over 30 mph on a descent, so fast riders will obviously blow past me. I used to start using the brakes to make sure that I didn't go faster than 25, so it is all up to you, and may change with time.
Not everyone rides to ride fast.
