I've had a number of commutes through less than desireable neighborhoods. On my last one, at the corner that I thought was the worst, one night I had a flat. As I sat in the light of the McDonalds parking lot patching the tube, someone came up and tried to offer me a $20 bill.

I've had trouble in good neighborhoods as well as some in worse neighborhoods, but as Chipcom said, pretty much a bicycle is too small to rate goofing with. So be aware and keep your distance from people and you should do okay. If you sense trouble, double your cadence.