I would say, you NEED to have the ID of the stem handlbar mounting bore and the center of the handlebar OD match. If the bar is too big, the stam clamp should fraccture as you tighten the clap bolt. the bar is too smal for the stem, you will over tighten the clamp screw and possibly never achieve tension. And due to the over tightening, it could still fracture the stem.
When the stem fractures, if you are riding, you will most likely crash and hit the ground, possibly with your face.
There are many sizes for the parts in question, and if they are now distorted, they are already over stressed and hence dangerous. Aluminum does not like to be cold-set.