This is certainly true if you HAVE goals, and you may want some, but you don't have to be that serious--there's nothing wrong with riding for the sake of riding, or riding for fitness without a specific aim.
Having said that, though, if you DO have a goal, you probably won't achieve it without specific training. All the distance riding in the world, for instance, won't make you a good sprinter. I enjoy distance riding, and I've done many centuries in relative comfort (key word is "relative"--they always hurt). But I'm a terrible sprinter, because I don't like to suffer and don't put in the time to get better.