The type of glue to try is super glue, aka model airplane glue, aka cyanoacrylate (sp?). Use the thinnest stuff you can find, as that is the strongest. It can be purchased at model aircraft hobby stores. It has a short shelf life (around 6 months), so get fresh stuff, not what has been laying around in your toolbox for 5 years.

I acutally doubt it will work for you, but it is worth a try. I have removed parts using this glue in the past but those parts had a large gluing surfaces. Use the glue very sparingly, just put the smallest possible drop on the broken ball driver. I've seen machinists use super glue to hold down a part that is being milled. The glue can be tremendously strong, but the surfaces have to fit together perfectly to get strength.

Superglue can be removed with acetone.