Just take a ratchet extension or something, push the spacer over and tap it with a hammer. Alternate sides you hit so it works the bearing out. Don't hit it too hard, just tap it firmly.

it's pretty normal for a headset to drop in on an integrated headset.