Everyone is different, so listen to your whole body, not just your butt or your legs.
The numbers quoted above are staggering compared to my first rides. I managed the 0.67 mile ride home from the LBS with a rest stop in the middle. For several weeks I needed to stop halfway on my two-mile commute.
These days, I stretch my commute out to 17 miles going to work, and shorten each end a bit for 13 mile rides home. When I get out with the club, I look at their 30-50 mile rides, then ride the 10 or 20 to the ride start, then again back home.