The speed demons around here are impressive, but I'm not sure they're helpful to
VegasJen (aka OP). Let me offer a few ways to approach the speed fear:
First, don't push yourself. Remember you're riding to have fun.
Second, make sure your bike is in good condition. Tires and brakes have enough rubber, derailers are adjusted, chain's lubed, there's nothing likely to go wrong that will interrupt the fun downhill you're going to ride.
Third, as you approach a hill, relax. Start worrying about what might happen and it'll kill the joy. Instead, go into a downhill with an attitude like, "I earned some fun by climbing that sucker to the top, now it's time to pin my ears back, grin real big, and fly down!"
Fourth, ride appropriately for the conditions. Smooth, straight road, light wind? Let 'er rip. Bumpy pavement and gusting crosswinds? Ease up. Remember #1.
Fifth, take your time (weeks or months). If you're comfortable at 30 mph, ride enough at that speed that you feel like you can try 33, or 35, or 39, and be comfortable at 30 if you need to back off. When you start feeling comfortable with a little more speed, make that your default "slow down to 35 and I'll feel safe" speed.
Finally, take some comfort that you won't automatically die if you exceed some arbitrary speed limit. Go back and read this thread as a series of testimonials -- like all us braggarts, you can learn to ride faster than your current comfort limit and then you get to brag, too. And remember, we're bragging because it was fun!