I have never experience this issue on my XTR rear derailleur before. I took it off in order to fix a bent hanger and after putting it back on I can't figure out whats going on. When I shift into the smallest cogs the derailleur gets too compact causing the chain to hits itself. I've tried adjusting the B screw with no luck. Am I missing something obvious?
I never learned to ride a bike. It is my deepest shame.
When you are mounting a derailleur that already has the chain running through it you have to twist the derailleur back quite far to get the b-tension screw to catch its tab... much farther than the derailleur will sit once it is mounted.
Three things you should do before mounting a derailleur with the chain installed are remove the rear wheel, back out the B-tension screw, and take the chain off the chainrings in the front... this should minimize the amount you need to turn the derailleur and give you the slack you need.
Or you can break the chain and install the derailleur sans-chain, then reassemble the chain with a new quicklink.
When a derailleur is removed the top knuckle can rotate. If it isn't spun back so there's spring pressure, it will exhibit the symptoms you described.
You may take a look at the steel plate that mounts directly up to the hanger. It will have a small detent tab that hits a corresponding tab on the aluminum derailleur body. If the body tab looks gouged, it may have over-rotated. If you can spin that tab back into place, you may be able to use a punch to bend it down again where it will hit the derailleur body. If there's not enough body tab left...it's new derailleur time.