I recently acquired a 1989 Trek 660 with a mish-mash of Shimano components. I'll list the setup first and then ask the questions:

Front Wheel: Hub: Dura-Ace, Rim: MA-40

Rear Wheel: Hub: Shimano 600 FH-6400, Rim: Mavic Open Pro

Front Derailleur: Shimano 105 FD-5503

Rear Derailleur: Shimano XTR RD-M910

Cranks: Shimano Ultegra FC-6503 (triple)

Cassette: Shimano XTR CS-M900 (12-32, 8sp)

...So, I've been having problems getting it to shift properly. This is my first bike with SIS that hasn't been a complete gruppo. Will XTR on a roadbike work? Do these Frankensteined components somehow make sense to you on paper?

The indexing is in the shifters themselves on these. What are the shifters?

the shifters are Shimano RSX brifters

Well, the front and back hubs don't really matter in this question... the FD probably doesn't matter either. XTR will absolutely work on a road bike. I am using a Deore DX mixed with 105 and Sora STI shifters on a road bike. What problems are you having? How many speeds are your RSX shifters, is the front shifter made for a triple?

I was telling more than necessary because it's such a funky stew of Shimano stuff. The whole range, as it were- RSX, 105, 600/Ultegra, and Dura-Ace.
They are made for a front triple and a rear 8, and that's what I have. I think I just need to learn how to adjust the indexing and everything should work. Or so it seems.

Yeah, I imagine that setup should work. My recent STI build was my first and it took slightly more work to get it dialed in. I am using 8 speed shifters with a 7 speed cassette so that added to the issues. I wasn't ragging on you about the extra info either. Extra info never hurt, we usually don't get enough.

Well, thanks a bazillion for all your info and help. Now I just need to start dialing it.

Are you sure this is an 8-speed setup? The Trek originally came with 126mm spacing, and 6400 is 7-speed, so is RSX if I remember correctly. Are you getting 7 clicks up and down with the RSX shifter? Perhaps you just need a 7-speed cassette. If the Trek was re-spaced to 130mm, dropouts and derailleur hanger may need to be aligned, as this can be the source of bad shifting.

7 clicks would indicate an 8 speed shifter right?

OK...lets claify a few things

Rear hub: 7-speed. Screw on outer (small) cog only.
Shifters: RSX STI
Rear cassette: XTR 8-speed

There is no way....its not possible, for the 8-speed XTR cassette to fit onto the FH-6400 7-speed rear hub without modification. Either the cassette is cobbled or the freehub body has been changed. On paper everything may look compatible but in reality parts from 4 different groups are being used. There's no mention of chain or cable type either. There are too many variables in play....