That Cane Creek is a cheapo. I started having a bunch of trouble with mine (living at the beach + cheap unsealed heatset = bad idea) and recently decided to replace it with something sealed.

If it's actually loose and you can't tighten it down all the way, you may need an additional small spacer.