Coming down Torrey Pines on Sunday I hit 37mph. Other than being cold I wished I could go faster. I've hit 47.3mph going down a hill near my house that I ride fairly regularly, it's steeper than Torrey Pines but not nearly as long.
You get used to it. As you ride more you'll gain more confidence on the bike and those high speeds won't seem so scary anymore.