FBinNY 

Obviously we can't hear the chain over the internet but here's a few things to check.

First there's the normal purr or buzz of chaindrive, who's volume changes with chain lube and chain/sprocket condition, but never gos away.

I'm assuming you have worse than that, so the next suspect is bad trim. IMO this is the most likely issue, since you have asymmetrical shift response.

I always have problems when folks number gears, whether they're starting from the outside or inside, nor do I know for sure which way the RD us sprung (98% are sprung to the outside or high).

So assuming that you have a typical high-normal RD which springs to the outside, and it shifts fine toward low, by cable, and sluggish to high by spring, than you have the classic sticky cables. This also cause poor trim following upshifts, while rarely causing trim problems after a down shift.

There's an easy test to confirm the diagnosis. Following a problem upshift, while the RD is still noisy, firmly pluck the bare wire away from the frame like a guitar string and let it snap back. If guitar plucking solves or improves the poor trim, then it's 99% confirmed that the cables are sticking. Sometimes you can fieldstrip, lube and reattach the cables, but just as often the housings are worn at the insides of the curves and the only remedy is to replace them.

Other causes of chain noise include poor/no lube, worn idlers, worn sprockets, mis-aligned RD hanger to name a few. But the asymmetrical shifting is sort of a tip off to check the cables first.