When I'm out riding maybe 1 in 8 people that pass me warn me first. It does not bother me if they are moving just a little faster than me because I can normally hear them coming. However, if they are moving fast enough to blow past me like I'm standing still it's annoying. My problem is if I don't know they're coming and think I have the lane to myself and drift to the left a little we have a massive wreck.
After being frustrated with this for a while I decided a few things:
#1- Get faster.
#2- Put a helmet mirror on and check your blind side once in a while.
#3- I put a bell on my bike to warn people I'm coming.