I think one of the problems is when you ride in groups frequently with people right up against you that you forget that most riders don't feel comfortable with people passing that close.
It's not wise, though, overtaking a rider that close as you never know how they're going to react if they're startled, and can easily swerve into your group and hurt you as well as themselves.
However, I hate to say this but with some riders I don't announce when I'm overtaking them as I've had a lot, and I mean A LOT, get startled when I say "On Your Left" and immediately swerve in front of me. I do give a wide berth when passing, though, and if that means I ride behind them for a while if I can't pass them with a few feet of clearance. If I am going to announce, I'm usually a good distance back so that I can see how they react and brake accordingly if they freak.
I rarely get passed by someone that I don't see coming, though, so it doesn't bother me if they are silent.