I use a power meter, but I still use average speed as a very reliable source of progress. I have a 30 mile long MUP I can ride on every day with only a single intersection and very minimal bike traffic, and the wind in SoCal is about as predictable as it gets....ocean breeze 90% of the days peaking at 1-3pm.
I train exclusively by watts, but many of my "goals" are based on average speed while others are purely about watts. Having avg speed goals lets you factor in aerodynamics (evolving your body position mostly) as part of your progress.
I could graph my avg speeds only for my daily workouts over the last 2 years and it'd be a gently upward sloping line (started around 17.5mph for 20mi, now up to 22'ish for 20mi), with probably a few outliers for those rare days where the wind was extremely high. From a day to day basis my avg speed will fluctuate up and down by 0.5mph or so, but smooth the noise out and it's a nice progression over time.