More problems with the sidewalk:
If there are driveways, people may have their cars parked over the sidewalk, forcing you to stop and walk around their car.
It's often difficult to get the attention of walkers from a distance. Expect to slow down a lot when going past walkers.
People walking their dogs are a greater nuissance since you don't want to run over the leash and choke the dog.
Something is different about sidewalk/bike-path asphalt that makes it difficult to maintain high speeds.
Sidewalks typically don't get salted in the winter time.

I try to use neighborhood routes so I can feel completely secure using the street. Map websites like google maps can help you plot out such routes.
On main roads, I felt more secure after I perfected my backward glance. If I'm going fast enough, I can get a good backward glance without swirving. Maybe this was only a problem because I wear a backpack...