It's difficult to figure out possible fit issues online(especially so without photos), but the hands going numb when you're on the hoods seems particularly odd. It's been my experience that the hoods are the most comfortable position for the hands, even moreso than flat bar.

Is your balance on the bike so far off that you're carrying a lot of weight on your hands when you're on the hoods? I wouldn't give up and switch to a flat bar before trying to figure out if the problems can be solved. You could try flipping your stem up if it isn't already. Can you move some spacers to under the stem to raise your bar a little? Maybe try a shorter stem? Have you tried moving your saddle back a bit(that can relieve excess weight on the hands, too.)

I converted one of my road bikes to flat bar just to have something different to ride from time to time, so I'm not someone adamantly against the idea. Just seems to me that you may be giving up too soon.