When I had issues with the chain dancing between several cogs it turned out to be my cable binding badly. One of the other symptoms was that it wasn't responsive to shifting to the next higher ratio cog on the rear and during the time it was taking to make one shift, I might have made several shifts of the lever so it's dance over a long period of time until it finally caught up.

New cable guide under the BB fixed the issue.

Though if your problem is similar to this, then I'd think that when you shift to lower gear ratios that you'd be having to use a lot of force on the lever.