The key to improvement in any sport is training yourself to stress yourself at different levels. You have to ride with fast people to get faster. You have to ride with slow people to get ready to ride with fast people.
It would be best to ride with both groups alternately.......