I'm feeling even more vindicated now. I had my professional fit yesterday. He actually raised my saddle 1.5cm. This is only 5mm lower than where I had originally moved it to by myself. So I'm doubting the extra 5mm that I had it raised caused much damage.
He also adjusted my cleats a bit (moved them back) and moved my bars up a bit. He said I could easily drop my bars back down but he wanted to see how I did with them up for a little bit.

Did a short ride after the fit last night and felt really good. Going on another ride tonight.