You are most likely pushing down through your pull to help your breathing. Very common in swimming. Key is when hand enters the water to make sure you have rotation to your left side sweep the arm from your elbow to your hand towards your hip then finish your pull. Some good drills to learn swimming with your hole arms is to swim with just your fist. Concentrate on using your forearm to pull the water. Also another thing is to not think about pulling your hand through the water but rather anchoring your hand and pulling your body by our arm.