This topic has been covered a lot on the Clydes and Athenas forum, although offhand I can't remember the exact thread.
Basically, the suggestions boil down to:
- You need to be riding an equivalent bike. In other words, if he has a decent quality road bike, you should not be trying to keep up with a heavy cheap mountain bike.
- If he knows he's faster and he really wants to ride with you, he should let you set the pace, or at least double back to join you if he drops you.
- If you have the room and the budget, consider buying a tandem. You will always be able to keep up while on the same bike!