@FB, thank you for the idea I will be trying that once I get home.
@HillRider, yes the stem and steerer are the same size, both 1 1/8".

The only other idea I had was to take some 400 grit sandpaper and sand the paint from the inside of the stem.