I don't care if anyone passes me by. There are a lot of people on light singlespeeds with little cargo that probably ride 2-3 miles to work. I carry full change of clothes, few gadgets and ride 8 miles. However, often I try to keep up with a stronger rider for the exercise, particularly climbing over the bridge - I often lack the motivation to press hard in the morning. Still, out of the heavy commuters I see (panniers, bunch of lights, etc) I'm usually the fastest one but I'm not getting jealous, I know my limits, and I ride for fun.
During my years riding with the bike club I met many women older than I am, who were not into racing on anything competitive yet they were in much better shape and were stronger riders that I ever was. It didn't hurt my ego, I actually befriended some of them and did many rides with them exactly for that reason: nobody was getting mucho. Riding with stronger guys often intimidated me as guys like to brag a lot. I felt much more relaxed riding with these women and improved a lot during that time.
Adam