Way to go!
3 1/2 years ago I first began riding road bikes. Average speed was 15-16 mph the first year, 16-18 the second year, and 17-19 last year. Between years two and three, like you, I found I was stuck. Asked the owner of my LBS, and he said "Intervals ... they work."
And they do. There were a couple of solo rides last year at 20 mph and 20.5 mph. They were 20 and 30 mile routes. Rolling, but not super hilly or very flat. I was also doing intervals those rides, and not thinking at all about average speeds.
Now I'm struggling with a minor knee problem, haven't ridden much this year, and feel very old and fat. Not even thinking about intervals at this point.