I really don't understand how some folks cannot manage anything as basic as riding a bike.
I used to do a group ride which used part of a local MUP, and sometimes used it for solo rides. In my time I have passed many people and startled more than a few. Never had a bad response, even from people who got scared ... they never got angry.
You guys are doing Something wrong.
Life is hard enough without wanting problems ... and you might claim you don't want problems, but still, you always seem to have them. I really don't want problems, and somehow ... it works out.
Maybe try meditation or something.