I average 4470 miles a year (past 11 years), but this year a surgery layoff means I'll be lucky to get 4000 in. Old age catching up!
Every year on my birthday I ride one mile for each year above ground (grim humor!). This year I did 73 miles, and was a half hour faster than last year, so there's hope. 4:01 minutes per mile average this year.
I don't think I'm losing too much fitness, if any. Six weeks ago I spent two weeks bikepacking in England and Wales, riding those darn 15-20% steep hills. I managed way more effort than I though I had in me. Sleeping in a small tent in the rain is getting old though.