Sometimes it's not the absolute speed that's scary, it's the acceleration. Descending with that feeling of really speeding up quickly can be a little freaky. Maybe it'd be better if you could find a nice staright downhill with little traffic, then let yourself speed up a bit, then brake, then speed up again, then brake. Braking all the time is bad, as is letting yourself bomb down the hill out of control.

It's not one or the other. Practice helps. Go have fun!