I've built a lot of chopper/tall/freak bikes that had some pretty shoddy construction, but this is one thing that I would not do.

You can't just get away with a fit like that. .6mm isn't a small difference when it comes to parts like stems which are designed and machined to very specific tolerances.

Plus, it isn't going to fail on a leisurely ride down to the beach, it will be when you are cranking on it and will probably end up face down on the tarmac with your bars in your hands.... wishing you would have listened to the advise given here.