I like to call the Venice Beach area of the Santa Monica Bike Path "The Freak Show". I've been there at all hours of day and night in the Summer and Winter, and have never had any trouble whatsoever. There is almost always a Police presence somewhere nearby because of all the homeless people who are on the border line of belonging in a mental institution.
If you want ride at a good pace, don't even bother with this path during beach hours. At all times, you have to keep a sharp eye out anywhere near pedestrian access points because tourists just don't know it is a bike path.