I applied WD-40 on both side. The rust is the WD-40 with some rust dust.

I will try to bold it in again and use my rubber mallet. But I don't recall hearing the bottom of the stem moving freely.