If they didn't invite you to hop on, you should have left them alone. That's probably why they were annoyed with you. If you can't restrain yourself, at least ask if you can join. As mentioned above, if the pace is too fast for you, take a shorter pull.