It wouldn't bother me either. Sounds like a good idea even. I sweat a lot and corrosion develops on the stem and spacers creating creaking noises and even corroding the spacer to the steerer tube. Chris King headsets have these thin plastic washers that help that situation but a plastic spacer would be even better I think.