I record pretty much every ride on Cyclemeter and/or Strava, sometimes switching to Wahoo Fitness to compare results. Even two mile errand rides.
I don't care much about overall average, junk miles, etc. Everything counts toward objectively evaluating my real world fitness. If I cherry pick my overall average would seem pretty impressive to me, but wouldn't help on the faster group rides I'd like to participate in once in awhile When my real overall average hits 17 mph, I'll know I'm ready to tackle those tougher challenges. Meanwhile, it's consistently around 15-16 mph, not quite good enough to hang with the local B group's 50 mile rides.
If I cherry pick only rides where I'm deliberately trying to ride fast, I'd overlook the errand rides where my average speed has increased from 8 mph in 2015 to 12-14 mph now. Since I enjoy casual group rides too, and used to struggle just to ride a couple of miles to and from the grocery store, it's the mundane stuff that helps show my real world daily fitness.
I even record and post my warm up and cool down stuff. If I cherry picked and cropped the first and last five minutes, my average would be much faster. But that would still be misleading because my warm ups and cool downs are also faster now.