So there is a lot of factors involved but will say this one up front, you can make a smaller bike bigger (longer stem, seat set back, offset seat post, wider bars...) however if your frame is too big you are very limited and kind of in a bad place.
Now I am not going to tell you which way to go, everyone has different opinions and what they prefer. Myself..I prefer a smaller frame than a larger one. I fall in between a Med and Large, I go with Med, I find it is easier to throw around, handle, this is for both Mountain, gravel, CX and Road bikes, I just picked up a Farley 9.6 last week (why I am on here looking for ways to slim her down more), and I rode Salsa's and Treks both back to back, I am a bit bias with Trek (own many Treks) but I really liked the colors, options, quality, cable routing of the Salsa, however the Farley felt very similar to my Trek Top Fuel 9.8 fit/feel/groupset..
Saying you 65" tall (5' 4" ) puts you on the shorter side, and would put you in a small frame, unless I read your size incorrectly, I would go back and give that small another try?