Yes, I notice the same thing. If you push down the reaction is to lift your CG. At some rpm you will hit the resonant frequency of bouncing off the saddle. If you equally pull up, then there's no net up/down force on your CG. If you only pull up, the force is down onto the saddle rather than off it. In any case, my theory is that since feeling the force on top of your foot is so rare, you notice it way more than the same force on your sole. Also, you usually support your weight with your legs and feet. Pulling up is unusual so you're more cognizant of it when you do it.