I see what you mean about the fact that they all have to pass you again, which can be annoying doing that back and forth thing. However, it's my opinion that cars pass me in non-passing lanes (ie: the two of us in a single lane) because it is inconvenient for them to remain behind me. I do not feel guilty about passing cars because it is inconvenient for me to remain behind them.
Make sense? I am careful about it; I always keep my eyes open for doors and cross traffic. I just don't think I should have to wait in a long line of traffic when I don't have to, and when they all just blew past me anyway.