Yup - the 'sleeve' sounds like a shim for using it in bikes with wider steerer tubes.

You should be able to remove the old wedge by knocking it out from underneath with the bike upside-down using a slim broom handle or anything similar.

You also might be able to get it out by somehow managing to screw the bolt back into it, and then pulling it up.

For future reference, don't undo the bolt all the way from the stem - it's not necessary to remove it - it just needs loosening before pulling it out.

You don't really want to be leaving the old wedge in there as it is probably in exactly the location that the new one needs to be in so as to be able to wedge in properly when it's tightened. It can be dangerous for the expander wedge to expand in the wrong part of the tube - it can crack the tube or steerer, and/or can just pull out suddenly whilst riding down the road. (As someone else mentioned - maybe funny in cartoons, but definitely not IRL)

You also want it far enough in so that it's at least at the min insertion mark.