I had the same problem. I was fine until a very scary almost crash going downhill event that spooked me. Then it became a psychological hurdle to overcome. Repeatedly doing hills over and over cured it. It's facing your fear basically, it just goes away. Assuming there is nothing wrong with your bike.
And while you are at it, ignore the internet tough guys that insult you because you are not as much of a man as they imagine they are.