Not at all if the guys right in front of you , but if he's half a block away and all he has to do is slow down to avoid a colision, yes to a certain degree. As a driver who practices defensive driving I am just a responsible in avoiding a collision as anyone else on the the road, no matter what I need to do to avoid it.
If a ped steps into the street and you can reasonably avoid him, should you? OR should you plow him over because he's doing wrong and you have ROW? See the difference?