As with most interactions, common curtesies usually work for most people. If you join a group, politely ask if they mind if you join, but be prepared to hear a polite no. If someone joins you unexpectedly, politely tell them you prefer that they not ride along. It really should not be more difficult than that.