On our group rides, we often come upon solo bikers or small groups, and as we go by, we offer a friendly wave but don't so anything not to spook them.
If you didn't want to continue riding in their group, why not just slow down as they continue their ride?
There was not need to panic, they weren't a motorcycle gang. lol