If you have a wide range cassette, ie. 12-32 or similar, the middle sprocket is in sort of a hollow because the progression changes there. This is how most cassettes are built, so the steps will be of a similar percentage. You can confirm this either by listing the sprocket sizes or placing a ruler against the cassette.

So when you try to downshift to the middle sprocket the chain may touch and want to shift to it's larger neighbor before touching the sprocket you want. A flexible chain, or having the upper pulley too far away can make this worse.

Before index, we used to add a shim to push the offending neighbor farther away, but that's rarely a good option with an index setup.

This is simply one of those things that needs to be dialed in, rather than painting by numbers and expecting the best.

Try adjusting the B-screw, replacing the jockey wheel with one with no float, or using an O-ring to reduce the float of the one you have. Shimano RDs are also sensitive to chain length, so see if it's better or worse when shifting while on the larger or middle chainring, and adjust the chain length accordingly (subject to maximum/minimum limits).