That wasn't my experience. When I started out, well, my first ride was home from the LBS. It's 0.67 miles at -0.5% grade (yes, slightly downhill). I had to stop and rest halfway. It was weeks before I could ride all the way to work without stopping to rest. The commute at the time was two miles to one job, three to the other. It was three or four months before I could "bang out 10 miles at an easy pace".
I appreciate you saying that as I was a bit shy to say that when I started out that a couple of miles was daunting... add in that there also was some break-in time for sitting on the saddle...