Wind resistance isn't the only thing to think about. Swept back bars also make your legs work harder. And they usually offer no variety of hand positions. As for numbness, that varies from person to person. You may find they're delightful even though flat bars don't work for you. Or they may be just as bad or worse.

If you change to swept back bars, it's more likely you'll want a longer stem, not a shorter one.

With all that said, it's not a crazy idea. You have to try it. Plenty of people have toured with swept back bars.