I confess that when I'm about to pass pedestrians that look, based on my extensive experience, as though they might harass, I put on a show of struggling very hard just to keep going. This seems to work, though it's more effective going uphill.