When you were younger you probably had absolutely no knowledge of your mortality, so you probably didn't even think twice about prying the stem as far as you needed to in order to change the bars, and on a steel stem, this is probably not too bad on the dependability of the stem...

I know I bent forks by hand to align them in the 70's, so do whatever it takes.