I would surmise that the presta valve adapter which has a rubber o-ring on it may be the culprit. I found them to be half useless and enjoyed my floor pump upgrade to a pump that works on both schrader and presta valves. If your pump hose is removable you may be able to simply buy a new hose with air chuck and continue using your regular pump.

Perhaps the oring on the adapter is failing which means pieces that air can get by the stem and leak out as you add air. I guess it would leak more as you were on the up stroke and then quit the leak as you were on the down stroke of the floor pump. Also there is a chance that due to using the adapter the presta valve is actually being pressed in by the adapter which can happen easily enough. I only used an adapter for about a week before deciding that it was way more of a pain than it's worth.