I commute through a couple less-than-desirable neighborhoods. Other than being a little more alert, I don't worry about it. The only reason anyone would have to give me trouble would be to mug me. But on a bike you're moving pretty quickly. An unsavory would have to see you, decide you're worth mugging, and attack you in a pretty short amount of time. It's more likely they'll find an easier target. Still, I give unsavories some extra room in the unlikely event that I need to maneuver quickly. I treat them similar to how I treat someone making a left turn and talking on their cell phone.