My bike is Trek Crossrip Elite 2014 and has VP-A61ACK headset on it and recently changed the top cap to "First" cap.
The problem is because of my new second-hand stem which is the Ritchey WCS C260 has a rounded hex bolt and very hard to take it out from the steerer tube and after I tried putting it back I guess I've done something to it. There is a gap between the fork tube and the (cup?) which is after putting the cartridge bearing and the compression ring and no matter how hard I push there's always a gap.It's more likely the cause of the compression ring, it didnt sit fully into the cartridge bearing. Second, after messing up with my stem I cant seem to tighten my headset enough, I tried loosen the stem bolt, tighten the top cap bolt and tighten back the stem. There's always a play where the frame connects the headset (front brake, front and back method).

Please help me out, thanks!

Your new stem is probably slightly shorter than the old stem. So when you tighten the top cap, it cannot pull up enough to tighten the bearing. You need to add another spacer underneath the stem so that the steerer tube is 3mm below the top of the stem.

What do u mean by slightly shorter? My old stem is 70mm and my new stem is 110mm. And do i must add another space underneath it? can i just put it above.

The steerer clamp may be shorter than your old stem. You can put the needed spacer under or above it as you wish. What you need to have is the stem or upper spacer 2-3 mm above the steerer tube.

To the OP, they are not talking about the length of the stem but the height where it clamps to the steer tube. Your new stem is probably a little smaller so it is not going to work without a spacer. To give you a visual, look at this photos taken from Slowtwich. The steertube must be 2-3 mm lower than your top cap as Hillrider and techsensei have advised. The spacer above or below is up to you as long as you have a spacer.

Alright, I managed to tighten it thanks to u all, I added another spacer above. tqvm