I think its just your comfort level. When you get in a car with a friend, you dont worry about them driving, but when you get in a cab, dont you wonder what the person is going to drive like? You know how your "normal" group rides and have a strong sense that they all know what they are doing, therefore you feel comfortable and can relax.
Next time you ride with the other group...try and spot some familiar faces to ride behind/next to/in front of...you'll at least feel more comfortable around them. Eventually, you'll get used to more people in the group.... Good luck...