Are you sure this is an 8-speed setup? The Trek originally came with 126mm spacing, and 6400 is 7-speed, so is RSX if I remember correctly. Are you getting 7 clicks up and down with the RSX shifter? Perhaps you just need a 7-speed cassette. If the Trek was re-spaced to 130mm, dropouts and derailleur hanger may need to be aligned, as this can be the source of bad shifting.