Preface: I don't consider my results to be particularly spectacular, or amazing. I am only citing it as a baseline for a common reference for this particular post. Obviously, any conclusions derived from a data pool of 1 is anecdotal at best.
2016
Distance 4,910.1 mi
Time 315h 51m
Elev Gain 174,265 ft
Rides 183
15.54 mph 35.49 feet elevation gain per mile.
2015
Distance 1,818.1 mi
Time 119h 2m
Elev Gain 43,163 ft
Rides 71
15.277 mph, 23.74 feet elevation gain per mile.
So, we can see here that in 2016, my average speed for the year has increased a measly .263 miles per hour. Super lame until you consider 1.5x the elevation gain per mile for a total of 4.5x the elevation gain for the year.
What isn't so obvious in the "average speed" question is all the other meaningful data. Despite 2016's seemingly marginal gains by that one metric, my average heart rate is easily down 20-30 beats per minute, for a given wattage, down 40 or 50 bpm for a given speed, heart rate lag and recovery are virtually non-existent, my vO2 max is 49mg/ml, up from the 20's, I ride further and faster with less fatigue, also my recovery between rides is better. I've only lost 15 pounds total weight in the last 18 months, but my body fat has decreased 11% from 27 to 16% (Meaning I lost 23.85 pounds of fat.) So that works out to be a 9 pound increase in muscle mass...and it's all in my legs.
What my average "moving speed" is, is anybody's guess. Flat/level ground with no wind and solo currently it's usually in the 20-22 mph range; For a given journey, lately it's 17-19 mph usually depending on terrain and how fresh I am. Factoring in stop signs/lights, rest stops/breaks, and all the other little things, we're right back to the 2015/2016 annual averages. (total miles/total time, and why the annual average increase of 1/4 mph data point doesn't really capture what it is I think you are looking to have answered.)
So, IMO, you are asking a well intentioned, but wrong question. Has your health and strength improved? If yes, then you are on the right track and stick with it.