It's likely not just the air you exhale, but the warm moist air rising up around your neck. I had similar experiences while hunting, and have since learned to use a scarf instead of a neck gaiter. If I get a bit warm or my glasses start to fog up, I open the scarf or take it off. I can also open my jacket to remove more warm air if it still isn't enough. This is for when it's not too cold.

For colder temps, 20F and below it gets more complicated with a neck gaiter or high neck heavy fleece, rolled down or up over my chin. Goggles and eventually a neoprene face mask for even colder temps. The face mask is the most problematic, but if I start to fog up, I just work on gettting the warm moist air to vent away from my glasses and goggles.