If I rode with my seat 1/2" too low, my right knee would explode. I have far too much experience with that through my own incompetence as a fitter/mechanic.
I don't use my dropper as much as I should, but some of the descents around here are a lot less sketchy with the seat dropped. I'm not convinced that only using it occasionally is really grounds for removing a post.