People hanging on won't slow you down so don't waste energy trying to drop them. In fact there's evidence they might improve your overall performance if they ride tightly behind. The problem is strangers can also cuase accidents.

To do 5 hours, you need a group that's fit and able to ride that long. I've been with people who can go 3 or 4 hours but really slow down after that. If you wait for them, you can miss your 5 hour goal.