Your big issue with just swapping the small ring for a smaller ring is that you'll probably be well out of your rear DR specs with any significantly smaller ring.

You'll have to sacrifice your top end by also going to a smaller big ring. But smaller on the top isn't really as bad as it may seem. How often are you truly putting out max power in the big ring and small rear? Might mean someone beats you to the bottom of the hill, but over the entire ride it probably won't make your times much different.

Knowing what kind of riding you do, your experience and fitness along with what your desires are might let someone help you figure out where to compromise.