Cars passing you usually wouldn't be a problem, although there are people out there who either don't care, or don't realize, that they drive by way too fast at too close of a distance.

Just be sure you can ride in a relatively straight path. If you happen to swerve, you can potentially get clipped if you do it as the wrong driver is passing.

Avoiding debris on the road can also cause this. Just be sure you can look over your shoulder comfortable while riding to make sure that it is safe to maneuver more into the road.

You should more so be worried about corners/driveways. Drivers don't usually look for pedestrians/cyclists specifically. You should always be ready to slow down before corners/driveways if you feel that a driver does not notice you coming.

Like people have mentioned earlier, the sidewalk is usually more dangerous. There are more obstacles to avoid, such as sign poles, telephone poles, light posts, electric boxes, whatever it is they put on sidewalks.

I usually only go on the sidewalk when after waiting for a red light, because I don't think cars also taking off will expect me to be on the road, and may potentially clip me. As soon as there is a safe opening, I get back on the road.