Of course, the group rides you would partake in wouldn't ride in pace line but more loosely spaced, also maybe two bikes shoulder to shoulder? You should be brave and join them but tell them of your worries, so that they would be aware of it, maybe you should keep in the rear of the group for the start.
It is well possible that the responsibility of riding in a group will impress on you, so that your problem or your worry will not materialize.