When I first tried to do this I found it very difficult to get the rubber bit out. What was suggested to me which did work was to undo the metal bit at the top, then put your finger over the hole and depress the pump, this should force the rubber bit out.
good luck
Daven