Newer derailer, with only 300 miles on it.
Put it on, adjusted, and it was working perfectly up and down the gears. This is 3x7 setup.
Now, there is an issue when changing to 4th gear, and 4th gear only. Going from 3rd or 5th takes much longer than any other. The only way to change it, is either wait, or push the gear selector little further, almost to the next gear. It's quiet when in 4th, no chain rubbing or anything like that.
Another adjusting is fixing the problem, but only for a very short time/distance.
Derailer hanger is straight, new chain installed the same time as derailer and cassette.

I notice it while riding on an unpaved, rough roads, but I had the same issue with previous Shimano LX derailer on the same bike. All cables and housings are also newer. The only things that I didn't replace in between the derailers on the drivetrain, are shifters and the rear wheel.
Bike was never dropped or damaged in any way.

What am I missing? Any tips will be greatly appreciated.

B screw adjustment?

I didn't touch B screw adjustment yet to be honest, but why everything works so well for the first several miles after each adjustment with L and Have screws, even without playing with B screw? Is it possible that it gets lose? Will try to play with it tomorrow.

Now I'm just a newbie and just started working on my own bike. I read through the idiots guide to bike maintenance and in the book they mention that the shifter and the gears have to be paired. The shifter is built to shift the cable a precise amount with every click. the gears are spaced to match that movement.

but if it works for a while and then stops working it sounds like something is loose.

Probably not helpful, but I'll say it anyways, just in case...

It's possible that it's not indexing correctly. Try letting off all tension and reconnecting the cable (starting over) and make sure the cable has a little slack. Shift to the next gear (it shouldn't shift) and then slowly backout the barrel adjuster until it is tight enough and shifts. Then try running through the gears. Always try the obvious and simplest possible fix before you start adjusting screws or worrying about bent/malfunctioning parts.

Is the shifter pulling the correct amount of cable? Deore should be a 9-10 speed RD, so you'd need to pull more cable for a 7 speed shifter.

If you have a wide range cassette, ie. 12-32 or similar, the middle sprocket is in sort of a hollow because the progression changes there. This is how most cassettes are built, so the steps will be of a similar percentage. You can confirm this either by listing the sprocket sizes or placing a ruler against the cassette.

So when you try to downshift to the middle sprocket the chain may touch and want to shift to it's larger neighbor before touching the sprocket you want. A flexible chain, or having the upper pulley too far away can make this worse.

Before index, we used to add a shim to push the offending neighbor farther away, but that's rarely a good option with an index setup.

This is simply one of those things that needs to be dialed in, rather than painting by numbers and expecting the best.

Try adjusting the B-screw, replacing the jockey wheel with one with no float, or using an O-ring to reduce the float of the one you have. Shimano RDs are also sensitive to chain length, so see if it's better or worse when shifting while on the larger or middle chainring, and adjust the chain length accordingly (subject to maximum/minimum limits).

I installed a "mountain" cassette for my vacation and screwed in the B screw.
Reinstalled my road cassette upon returning and forgot to back the b screw back out.
I had an issue with the middle cogs. having a "delayed" shift until I remembered to back the B screw back out.

I've had a similar issue before on different bikes after swapping derailleurs, cables and/or shifters. As things "settled in" or the cable stretched they'd start hanging up or skipping one of the middle cogs, usually 3 or 4. I've always been able to fix it with a minor trim adjustment at the shifter or derailleur, took a few tries sometimes but I always got it eventually.