I actually do use a layer that is totally non-breathable and waterproof whenever the temperature drops below freezing and have been comfortable that way in my nominally 30F bag down to 0F. The trick is to put this layer right against the skin so it holds in any moisture. As long as you're not overheated and your skin is moist, the pores shut down and you stop perspiring and therefore stop losing heat from evaporation. This 'vapor barrier' also prevents any moisture from getting into your bag so it maintains its full loft and insulating power. A secondary benefit is that I wake up less thirsty in the morning.

I actually came across this method by accident on a trip where the weather was much colder than predicted. Started shivering during the night and remembered that I had one of the 2oz mylar sheets along for emergencies. I wrapped myself in it and got back into my sleeping bag and was very comfortable for the rest of the night. Later read about vapor barriers and realized that that's what accounted for the added warmth. BTW, the mylar type is rather noisy and I now use a waterproof nylon liner instead.

But note that vapor barriers should only be used when it's really cold - below the normal comfort range of your bag/pad combination. If you start to overheat your body will start sweating like crazy trying unsuccessfully to cool you off.