Is there a bolt that secures the handlebars, in addition to the quick release? This bolt may need to be tightened, it may need a few washers, or both.
Try adding some nylon washers, these will slide easier than metal ones. You can usually get them at any hardware store.