Certainly I wouldn't start intervals until I was quite confident in my endurance, and had felt I hit a ceiling.
And yes, my livelihood doesn't depend on my speed, however the more I can ride in less time does help with getting more done, or more cycling in.
Thats a good point chinarider, that being able to average a fast speed does not make you any healthier.
I think what I've taken from this thread is that I will indeed hit a plateau where my speed does not increase with leisurely long-distance cycling. If I want to increase it from there, all I need to do is work harder during my rides, but be careful not to overdo it. I appreciate the help.