My hunch is that you get clammy because your outer layer isn't breathing enough, so your sweat has nowhere to go and you stew in it. (Not always, just when conditions aren't favorable to you, probably during more intense rides.)

If that's the case, moving the barrier closer to your skin seems like it will make the problem worse.

Some people use a wind shirt immediately next to their skin when they sleep (camp). You could try something like that for a ride, use your smallest and tightest wind blocking jacket as a base layer, then add insulation over it. See how it works before you invest in more gear.