Your age as well? If you are young and resilient in the back and neck, the lower handlebars might be fine.
If you are older, less flexible and prefer greater comfort, a larger frame rather than a smaller one is preferably because generally, it places the handlebars closer to the same height as the seat. It means you don't have to fiddle so much with stem angle, length, and height, seat setback, and likely pedal-over-spindle measurements.
There is also less seat post exposed (which can be an issue in flex and if the post is too short, damage to the seat tube).
My now ex-go-fast road bike was a smaller frame than my touring bike, but for various reasons related to fit, I wouldn't consider it for rides much over 160km (I could probably just get away with 200).
The bike I use for touring and randonnees is a much larger frame by comparison, and I can ride that all day, every day for over three days (1200 randonnee) without too much damage to the body.
As sivat says, larger is a better option for more immediate comfort, providing it meets the other requirements in standover height, toptube length, headtube and seattube angles, BB stiffness, etc.
But, if you regard 4-60 miles as long distance and are happy with what you have for its specialised characteristics (hill-climbing) and comfort levels over rides of that length, then stick with it.