The only portions of 'rough' neighborhoods that cause me real concern are places where the road narrows down and a lot of kids hang out during the day. There was one spot like this on my route where I had to be extra alert as some of the 'kids' thought it was funny trying to make me crash. There just wasn't that much space to manuver.
The sad thing is this will no longer be an issue as all of the houses along that little strech (where they all lived) are now boarded up and I presume foreclosed.
The other thing that gets my caution up a bit (and it is most likely just me being a worrier) are Friday afternoons in the summer. Lots more people out and a bit more uncertainty.
I remind myself how silly this is by thinking of the day I crashed in a 'bad' area and was waiting for my wife to come get me. I stood there watching the cars fly by and maybe 20% were not talking on cell phones --- for people in cars attention to the road is for suckers, I guess.