There are not that many riders who would pass me on the flat open road. On the bike path it is a different story, I'd say about half who pass me there are jerks. The other half are simply going faster when I'm enjoying the scenery or keeping a steady pace (E.g. not speeeding up when thgnis are more open. Some days I would speed up when safe other not).
Serious climbs are a totally different world. There are very few riders who do not pass me on hills. Not to say I'm slower than everyone, just that anyone who can even hold my wheel on hte flat is apt to pass me going uphill.