I'd probably go ahead and remove the fork.

The first attempt might be to simply invert the fork, and try to fish the wedge out with a spoke.

You could try to thread a bolt onto the wedge, but you might need a longer than average stem bolt. Still, something could be improvised.

Probably not a lot you can damage as long as you resist the urge to do a lot of pounding.