When I ride with someone we talk about this beforehand - what to do if one person wants to ride faster. We usually agree that's fine, but set rendezvous points where the faster person will wait. If this isn't agreeable then it's better not to ride together. Why annoy someone?