Are you sure it's not because you're moving so much faster than the bums? It's much harder to accost a cyclist that it is a pedestrian.

Here, many of the bums are on bicycles. They seldom confuse me with one of their own. First I wear a helmet instead of a dirty knit cap. Second, I have halogen headlights and bright blinkies, rather than broken reflectors. Third, I carry stuff in panniers rather than hanging plastic bags from the bars. Finally, my cruising speed is considerably higher than their wobbly average speed.

Cagers certainly don't notice the differences, but the bums immediately recognize that I'm outside their hierarchy. Which only confuses them when I acknowledge them the same way I do any other cyclist.