I've done a 64km (924m elev) ride and just did a 19.8km (580m elev) ride. The first took 2:30hrs the later 1hr. I thought my lower back was sore because I have been on the same position for a while during the 2:30hrs ride but I realized I felt it a lot during ascension during the 20km one. The 20km ride was 5 climb of a small mountain near my home in town. I might be moving my back a bit too much when ascending.
I noticed I was pedaling slower on my highest gear than some folk who were passing me. I have no other option but to swing a bit every stroke which might cause too much movement on my back. This is just an assumption, I wonder if anybody have an other suggestion that could help me alleviate my problem?
I do "a lot" of ascension, about 800-900m every ride. I try to stand as much as I can to alleviate some other problems (bum soreness).

I did the bike fit myself. 145 knee angle and aligned with the pedal:
BikeFit - Road Bikes
It helped a lot but I guess there's something left to do!