The Park Tool pages go step-by-step, and don't expect any previous experience. I go here first when I'm doing something new.

Don't just experiment with the limit screws. Set it up correctly.

This is the way to do it. Start from scratch: disconnect the cable and adjust the high and low limit screws correctly, then do the cable shifting adjustments. And it's good to know all these steps for when you replace cables, anyway.
A bad setup will allow the chain to shift past the largest cog or allow the derailleur arm to catch on a spoke--that rips the derailleur off the bike and usually damages the frame, too.

Fraying cable
My experience is that a fraying cable doesn't shift easily to a smaller cog, it kind of hangs back and requires two shifts smaller then one bigger. If the cable is this frayed, it'll break very soon. It might be slow or off center when shifting to a larger cog, but that should occur on most or all of the cogs, not just the largest one.

Cables fray within the shifter where the cable makes a bend, that flexing eventually starts breaking strands in the cable.