Thanks for the help everyone.
Sixty fiver, I guess I would agree with you, and after doing it this time I am sure that next time I could do it in at least half the time. Figuring out where the bolt was on the brake, and realizing that I had to disconnect the brake cable to get to it took some time, but the rest went well.
In fact, I did try to turn the nut on the bar end shifter the wrong way, and I was thinking that it was unnecessarily tight. Luckily I didn't strip it or anything like that, and finally got it out. Next time I will shift to the slackest position as it was a little hard to reinstall.
Hopefully this stem will work out! Previously it had the shortest nitto tectonic stem on it, and now I am more stretched out and don't feel as cramped. I'll see how it works out tomorrow when I have to go for a longer ride than just around the block.