Envision something for a moment. Take two random motorists. Put each in a car one mile apart facing each other. They must each accelerate toward one another to 60 mph and pass one another with 4-5 feet. I doubt most of us would be willing to risk that and we'd be hard pressed to MAKE ourselves pass another car at an effective speed of 120 mph within a few feet. Yet we do it every day and the only way we are willing is because there is a little line down the center of the road, "to keep us separated". With that, we have a boundary, gives us confidence to do something no way we would do otherwise, and it allows us to do it in relative safety.
There is no such "boundary" that separates a driver and cyclist. So a close pass certainly increases the danger of a possible miscalculation. That's why I'm thankful that in NC just last year they passed a state statute requiring 4' of passing room. This will HOPEFULLY be enough cushion for the ones that can't tell 4' from 2'. The REALLY BAD news is that like most new state statutes, it was never widely publicized and I doubt seriously most motorists are even aware of it.
Frankly, I don't want to be passed at 50mph with 4' between me and the motorist. But I'll take that over a slow speed crash any day.