I put 2 squares of electrical tape over the "tabs" on the pump handle. Duct tape would probably work better. This provides just enough friction that it won't come out accidentally, yet makes it easy to use when needed.

Quite literally a half-cent solution.