If you never use the drops, and you don't need them, cut them off. Flop and chop the bars.

I think you should change the bars, because you want to. But I also think you need to check your fit before you make a disappointing change.