Yes, Presta valves can be funny that way.
Push the valve as deep into the pump head as possible. You might have to hold the pump head while you pump. If you start with a flat tire, you sometimes have to support the back of the valve by pushing on the tire behind where the valve is.
Good luck. You'll get it.