"Friendly fast" can be deceiving. If the ride is designed to regroup, they expect to wait for people. If it's a race pace from start to stop, they won't wait and you should know how to get back home. If for some reason, you feel that you are holding them back too much at regroups, you can make up an excuse for them not to wait any more (i.e. "I'm taking a different route home").