Couple of ways of looking at this.

If you want to increase speed or endurance- go out with riders that are a bit faster than you or do rides that are longer and at a pace that is just above your level.

Or you are trying to achieve what is not possible.

Depends on whether you enjoy the rides or not.

A 20 year layoff does mean that you have not got cycling fitness back yet and it could be some time before you do regain the fitness you think you should have. I was still running competitive 15 miles cross country 30 years ago but I doubt I could run a mile right now. I could train to get a 5 mile run in- but not at the pace I was way back and I doubt that it would only take a year to manage that.

I would suggest going with the club once a week till you build a "Base" back again and 170 miles a week will do that. Just don't do all the rides at a level that pushes you above what you can manage.