I never, ever have been passed by a low-income commuter or just plain vagrant --judging by their disheveld appearance, smell and amount of crap they are lugging. And I see them a few times a week on this one MUP that goes underneath a highway near my house.
I have, however, been on a MUP and someone decided they wanted to race me. It's not the car-free commuters. It's not the homeless. It's never a woman. It's typically not even a young man.
It's a dude in his 30s like me, or older. I just ride my pace and ignore them.