I have no evidence, but I suspect real slow bike commuters who are on the road attract a lot of ire for being on the road then faster commuters. If you are going very slow, your speed makes you appear that you should be on the sidewalk to the cagers (regardless of whether that is true or not).

I think if I dress the part (bright colours, helmet, blinkie, etc.) I get more respect more then anything to do with my speed. And while I'm sure greater visibility is part of it, I don't think it is the whole thing.