A couple of things you need to understand.

First of all, it could take years, (yes years) for you to build up all the aerobic potential you could have. And you can build this aerobic potential - simply by just "riding around" - without trying very hard.

Two - you simply "MUST" learn how to deliberately ride slowly - just so, you can learn to "race" your bike when you have built up some "pent up rest miles."

Hope these few lines makes sense. But it really is the whole story....