I see the appeal of hanging it up, hitting the gym, not having to be out every weekend morning, not driving to races, etc. I've been tempted to call it quits a few times. Personally, I'm not quite ready yet, and what holds me back is the thought that I'm probably not going to reach this level again if I stop.
My tentative plan is to finish this season as well as I can. Hopefully we will have a kid on the way soon, upon whose arrival I'm prepared to take a step back from competing. I think I'll focus more on the gym while still going on casual rides at least once a week. Maybe commuting if it makes sense. Then assuming we settle into a good routine within a few years, I can think about coming back, possibly focusing on crits and track.
If we don't have offspring then I think I'll take a year off to do big fun rides, hit some dirt and gravel, hit the gym, run, but not focus on training per se. I'm with you on your analysis of muscular losses. I've managed to get fitter but I do think putting on some additional muscle would help.
Anyway, not really advice, just wanted to share that you're not alone with these sorts of thoughts. It's a hard sport, sometimes a break is good.