Bummer. I felt really bad for losing a tool roll last year (left it on my car's hitch rack and drove off) and that was a minor thing compared to yours. How did you not hear/feel the loss of the pannier? I've had a number of fall-off events while riding and I was able to hear almost every one.
I hope the finder contacts you. Since you lost your phone, it may take a while. I found a phone once and was only able to find the owner when someone called it.