I usually filter up along the right, and stop next to the foremost car if it's a straight through only lane, or just behind the front car if they can make a right turn.
I would NOT do this if I had to take the lane on the far side of the intersection, as you're right, it wouldn't be fair to the drivers, but if there's enough room for them and for me, then it doesn't seem any different to me than when they passed me (IF they did...) earlier... different vehicles, different speeds.
If there's a right turn lane, I travel on the line between the right turn lane and the straight through lane.
I do always slow WAY down... to about 5 mph ... ESPECIALLY if I'm passing between two lanes of stopped cars. I want to be able to stop in a big hurry if someone suddenly decides they aren't in the right lane, or if someone opens a car door and jumps out.
I only take the lane in a couple spots on my regular commute... The roads I've chosen are fairly lightly used, and for the most part plenty wide. (They just widened one part where I used to take the lane... added almost 3 feet of shoulder! Pretty sure they're going to paint it as a bike lane.
)