Work hard and you can expect that in 2-3 years you will hit some peak shape... improvements in one's cycling take time and miles and it never does get that much easier as once you hit a plateau you seem to kick it up another notch to keep the challenge and adrenalin flowing.
My ride out in the morning is leisurely as I ride with my daughters who can cruise along at 20km / 12 mph and I just gear down and spin to warm up for the legs for the ride back to the shop... I usually pace cars on the return. I do the reverse in the afternoon when I pick them up and the ride home is usually a cool down although sometimes those girls like to crank it.
Was so busted up three years ago I could not walk or ride and it has taken some time to get back into a semblance of the old shape I was in... last year was a good season and since I ride all winter I come into the nice season in pretty good shape.
When it gets easier you know you are either slacking or need to kick it up a notch.