I am mostly only bothered when I find myself in a situation with a lot of "fast" cars (40+) and no safe shoulder. Urban driving doesn't bother me because the cars are generally slower, and country riding doesn't bother me because the cars are generally fewer.
Suburban riding (at least around the commercial areas where everyone is flying down the road and swerving into shopping centers) is the worst.