As someone who's lapped a field before, it is just annoying to have guys out of contention in the way. Of course even if they finish ahead of you you still beat them (as the lapper), but at that point they do just get in the damn way.
That said, you don't need to do anything anyone besides an official says in a race, generally. So yeah guys will yell at you to get the **** out of the way, but it's up to you what you do next.
Yes, it's their "fault" they lapped the field and everyone is now mixed back in, but it's also your fault for getting lapped. Usually it's the case that some in the group want to lap (bc they have teammates in it), some don't want to lap, but can't control what happens.
I'm confused though. If a group of 20 lapped you, how did you get 13th?