Your friends are correct. If you really want to get faster and be a better racer, you need to spend more time building your base.

Think about a building. For a small house, you only need a small foundation - sometimes its nothing more than a few cinder block stacks. But for a skyscraper, you need a huge foundation, often deep within the ground. Strength and speed operate the same way. As your base grows, so does your ability to go faster longer and to put out higher speeds with greater frequency. Futhermore, the dips between your power will lessen.