OK...lets claify a few things

Rear hub: 7-speed. Screw on outer (small) cog only.
Shifters: RSX STI
Rear cassette: XTR 8-speed

There is no way....its not possible, for the 8-speed XTR cassette to fit onto the FH-6400 7-speed rear hub without modification. Either the cassette is cobbled or the freehub body has been changed. On paper everything may look compatible but in reality parts from 4 different groups are being used. There's no mention of chain or cable type either. There are too many variables in play....