There should be two bearings in the shifter body. One gets pushed in from the front. The other from the back. There should also be a spacer/washer between the two bearings. Make sure that didn't fall out and roll away.
Here's a photo of the rear one:



Are you sure this isn't a crack. Try to pull it open.


The cracked carriers in the photo I posted are metal, also. I'm told the plastic ones last longer, but I've broken one of those too.