Building up leg strength and cardio over time will give you best results. You can do that with shorter rides if you push your limits on them. As mentioned by others, keep mixing it up: long rides and shorter ones; but, my advice is to always try beating your time from the day before. Keep a log.

I've been at this for a while, and to be fair, it really never does get to where you won't feel a bit worn out after riding more than 25 miles. Under 20, and it's pretty painless. But, once you start pushing 30 and beyond, the hands tingle, the legs want to coast more often, and the hunger starts to kick in. I started taking gels with me and using one of those every 45 minutes or so. It helps. But, it doesn't make it painless.