You need to decide do you want to participate in all of the group rides or do you want to train.

Compare it to golf. My father in law golfs just about 2x every week during the season. But he never practices refuses to take lessons. He has been doing this for 50 years. He will then complain that he can't break 90. If he took lessons and practiced but only golfer 1/2 the time he would cut 10 strokes. But golfing with his friends is fun pracrice isn't.