The adjustment of your seat's position (or other things on the bike), the model of seat you're using, or both of the above could be to blame. Sometimes, a setup will be fine for 5-ish miles, but no good when you increase to 2 or 3 times that distance.

I've learned that some very small adjustments can make quite a difference, a seat that works well for you on one bike might not on another, and you have to be patient and persistent. Don't give up or decide you're just not meant to ride that much. There's an arrangement that will work for you, but you'll never find it by asking questions online or looking at hundreds of seat pictures. You have to go through the trial and error process.