you're damn right drivers are scared when they see bikes on the road. this is america: a nation of extraneous liability and frivilous lawsuit.
they are sitting behind a moving 2000+ lb. vehicle with some guy in spandex sitting on some tubes rolls next to them. it's not hard to veer left or right, to lose control, or for a biker to lose balance and fall. a car veers into another car: possibility that everyone will be fine. but a car veering into a biker, and you're a dead man.
drivers who stay back from riders or give plenty of space when passing are being respectful, responsible, and a credit to the cycling community as well as the community in general. Be considerate of BOTH worlds of vehicular travel.