I know it doesn't seem like a lot of space to squeeze down to but trust us, you want to use a shim.

Without a shim the bars end up only being clamped at a few points and that leads to wobble in the bars in the stem clamp area. Or in the case of a stem that doesn't use a separate cap it means you're stretching the metal a lot to get the stem to close down far enough.

In a pinch you'll find that a shim cut from the side of a pop or beer can will do nicely.