This is the sign of a mature cyclist IMO. I'm always like "Really, guys? Really?" when people get pissy about me passing them. It just doesn't matter. I go different speeds on different days and some days I might be taking it easy because of an injury, nursing a broken bike home (once had a frame snap on me on a ride and I limped home on a bike with a missing chainstay... I'm sure a lot of people thought they were awesome while they passed me), etc. So, just because somebody passes me doesn't mean they are faster than me.