Anytime you pass someone, assume they are on your tail until you do a headcheck that shows otherwise. Now someone catching up and sitting back there silently would be weird and dangerous, but I don't think that's the common problem.