I have this weird feeling that my Brooks saddle is too low, but if I raise it, I get the hip issue.

There seems to be a reoccurring theme here...