I've had that happen to me before and have ended up doing the same thing - you catch up to them, ponder about passing them, then just hang back. I just get scared of angering people if I pass them, I get paranoid about them yelling or something.