An hour of aerobic exercise per day will do your conditioning all sorts of good compared to nothing. Now if your comparison is with people who train much more and train effectively, an hour per day will look somewhat pathetic.

I would say though that an hour per day is enough to keep you in excellent aerobic shape if you train effectively. You will need to really push at times and know when to recover. That comes with study and experience.

The other thing is someone is always going to be faster than you are. It is a fact of life. Even on the tour de france, the eventual winner generally does not win very many stages.