Originally Posted by
FBinNY
Modern FD cages have highly shaped and stepped side plates based on assumptions of the chainrings they're being used with. Sometimes changing the rings is detrimental if they no longer matched the shapes of the cage.
But I suspect, that your problem may be that your 39t ring was made as an inner of a double. That means that it won't have any shift gates (cut down teeth) to start the derailing process.
Are you sure it's an XTR front derailleur? Fietsbob is correct that MTB front derailleurs are not designed for large chainrings but since yours worked with a 52T ring in the past I'm doubtful it's an MTB fd.
FBinNY is almost certainly right that your new 39T chainring was intended as the inner ring of a double, not the middle of a triple. So, if you are willing to "customize" it as he recommends, that should reduce or eliminate the problem. Shimano does make a triple crank that uses a 39T middle ring, the FC-5603 105 triple, and, if you can locate a replacement 39T chainring (part # Y-1GF 98020) it should work without any other modification. However, this is a 10-speed chainring and may cause chain rubbing if used with a 9-speed chain. Shimano's 9-speed 105 triple cranks (FC-5504) only came with a 42T middle ring.