You're fine. Presta valve only allows air to flow into the tire, unless you press the tip with your finger. It is possible the pump head will knock the tip a bit as you pull it off and let some air out, but that is negligible. The whoosh is mainly from the pump and hose.