Originally Posted by velociraptor

Actually you are not shifting down. Downshifting would be going from a smaller cog to a bigger one.

From your description, your derailleur is not upshifting properly. I know, it seems counterintuitive, but it's correct.

A possible solution:

First, make sure that the high & low screws are set to position the jockey wheels in their proper place directly under the biggest cog and smallest cog respectively.

Next, give the barrel adjuster at the end of the derailleur a turn or two in the direction away from the bike (clockwise). Retry the upshifts.

Rinse and repeat until it upshifts and downshifts correctly. A balance must be struck.

Adjust the B screw tensioner to add shifting crispness

More detail here.