To the OP. Falling off when you get tired is no big deal. Yes its annoying to the guys behind you, but they should not say anything because they've all been there and what do they expect you to do? I would agree with the previous posts. I don't like it when someone can't keep up with the line but doesn't pull out causeing a huge gap in the line before the next bike back realizes what's happening and can try to make up the difference.
As an example, the last time I was behind someone who couldn't keep up the guy immediately behind him refused to pass. So I had to pass two bike, then make up the two bike lengths lost before I realized what the situation was. Now I have to signal to the bikes behind me to follow me around these two guys. What a mess.
It would have been much better if he simply said "pass me, I'm falling off". We could easily make up one or two bike lengths.