Tough to quantify how much you would lose. You would be less aerodynamic, but you say you stay on the tops anyway with drop bars, so maybe not so much.
As I said, there was a guy who did quite well on longer rides with our club. I used to call him "Flat bar guy" before I knew his name. If I recall correctly he ran his bars lower than the saddle. It worked for him but to me the limited position would be frustrating. Whenever I'm cruising a mountain bike on the road I wish I had drops.