I'm a pretty good size guy. And I've been pistol whipped and robbed in Pittsburgh while walking home after work. Scumbag thieves only got $10 and my pride.
Riding at night is one thing, if you have proper lighting, its fairly safe. I find it very enjoyable and do it frequently.
But I'm behind the Orange Curtain and ride in Newport Beach/Irvine/Huntington Beach. Even there I've had a few run-ins with some fart-nozzles who just want to make somebody's life just a little tougher, Plus there is always the wildlife that keeps you on your toes. Lots of rabbits darting out in front of you, plus quite a few Coyotes. Plus some trails are corridors for the Homeless and transients (read: the criminally insane) and gangbangers.
Best advice is to ride with a trusted friend or group.