Is it the harder plastic that shatters without bending or the kind that will bend and deform a little?

If it is the hard plastic type I'd try and shatter it by prying inward with a small screwdriver or striking the screwdriver from behind to push the plasitic away from the threads. Do this until a portion of the inner threaded ring was opened and then it could be squeezed and pulled out or further shattered until all the threads were just small curved bits that would not stay pushed out against the crank threads.

If it is the bendy-soft type of plastic then I would take an exacto-blade and score the inside of the ring that is left and then try and peel it out -bending the ring into itself as soon as the scored part broke and the "C" that was left could be pulled inside of itself. Or just carefully insert a small flat screwdriver in between the plastic and the alloy threads and try and peel it inwards and away from the threads enough to cut or grab to yank out.

Plastic is not as hard as aluminum so if you just yank/pull/hit at the plastic it can't hurt the aluminum. The plastic "threads" should fail and strip WAY before the alloy ones will. Just be careful with any metal tool up against the alloy threads.

One other option is if you can get a narrow shouldered crank-nut socket in there past the plastic bits you might be able to remove the crank bolt and then use a regular "3-arm" automotive gear-puller to pull the arm off without removing the plastic first which might just pull the plastic out or at least allow you to work on the crank up on the vise.