I think this is normal. You'll get over it with practice. A good thing to do is find a road with a gentle grade and wide sweeping turns. It's a big plus if you can find one that doesn't have blind driveways or any driveways. That way you will have less to worry about on the way down. Once you find a road like this, ride down your normal way. Then try going descending again except using the brakes 30% fewer times than you did before. Keep decreasing it until you only haev to tap the brakes a few times to maintain a reasonable speed. What defines reasonable is up to you. Some people feel comfortable descending at 50+ mph, others don't want to go over 40 or even 30.
I took the same approach when learning how to ride without hands. I'd put my hands on the bars, then lift them off and clap once. I repeated this a few times. Then I'd do the same thing, except clap twice to increase the time spent off the bars. I kept doing this until I could ride without hands for longer periods of time. The basic idea is to just do things in steps until you feel comfortable.
It's good you brought this up and want to work on it. riding your brakes on a long descent can heat the rims which isn't a good thing. You can get tire blowout that way when the heated rims make the tire pop. I don't think this will be a huge issue on a 3 mile descent, but if you were doing a 15-20 mile one it might. Also, by braking less your hands and arms will be a lot less fatigued at the end of a ride.