I'd loosen the stem clamp bolts completely and wedge something into the gap to spread the opening slightly. That should loosen the adapter. Even then you may have to use a punch (brass, plastic or aluminum are better than steel) around the adapter edges to drive it out.