I commute in pretty hilly (though not terribly brutal) terrain just fine with a 46/16 after having been on the free side with 46/17 for a while. I do OK with no foot retention and brakes. (Yay Grip Kings!) I agree that the difference is more significant for downhill; anything less than what I've got and I'd be spinning way too much (or more likely, leaning on the brakes a lot more).
Don't be afraid to use the brakes! I've had to remind myself a few times that if the cadence is getting too much I can just brake--no need to be all macho about it and get ejected in the middle of traffic.