Update:
I realized that there was dirt/old dried grease in the shift lever caked very thick. I worked at it with a pick for a good while on each side and was able to force it into pieces. I was sort of right it was just stuck together from years of being in that state. Found the bad side is a little torn up near the band, going to try to clean and repair all the pieces and reassemble it now. Thanks for the help.