I tend to concur with Andrew.
However, shims are an option. You can buy specific shims from one size to another, that are easy to use because they have a lip to prevent it slipping down too far, or, more cheaply, experiment with e.g. beer cans, hobby shop shim stock, or hardware store sheet aluminum.