Google's already got a fleet of automated cars that have been racking up tens of thousands of miles. Not an expert on the field but they're using a combination of radars and image processing, and it's likely that those tech combined give you a more accurate vision of what's in front of you than what a person would see, even in something like a snowstorm. Articles from a year ago point out manned drivers had to take over in oddball situations like a bike running a red light, but these are safety situations that you can program for and avoid (in the case of a biker avoiding debris you could recognize the hand signal through visual sensors, or the movement of cyclist heading into traffic lane, or both). And the more automated cars there are in the system, the less these safety stop situations will occur anyway, as they'll be in constant check with each other.
May seem like a pipe dream now, but costs of auto accidents are staggering enough that it'll happen, and probably sooner than you'd think.
As a software developer I would find it a nightmare to program something that could reliably interpret hand signals even if we all used the same ones. Navigating through a snow-storm is not just a matter of being able to see, it's a matter of adjusting the way you drive to the road conditions.
If a driver has to "take over" during oddball situations that means that this driver has to be paying attention the whole time. If that's the case, it doesn't matter how many 10s of thousands of miles the Google cars racked up, you still need the driver.
Think about it this way. Toyota has run into all sorts of trouble and lawsuits because of a software glitch connected to their accelerators. It's not enough for it to work reliably 98% of the time. If Toyota can't get something as simple as an accelerator to always work correctly, I'm dubious about a manufacturer's ability to completely automate a vehicle.
Imagine that a software upgrade that introduces a bug. Instead of a driver making a mistake and causing an accident as an isolated event, now you've got the potential for thousands of cars to do the wrong thing under a certain set of conditions. The other thing to consider is maintenance. These Google cars are brand new. What happens to the sensors as they age?
Further imagine a terrorist bent on wreaking havoc. Presumably these cars communicate with each other and traffic control systems. What if these are hacked into? Didn't Iran bring down a drone by messing with its communications?
Sometimes more technology is not the answer.