It's either a fit issue or a bar issue. Plenty of people can ride bikes with flat bars for long distances, and plenty of people can't. You could have too much weight on your hands. You could be one of the people who needs a high, wide, drop bar on your long distance bikes.